PitchMeAI Jobs
Zafin

AI Evaluation Engineer

Zafin · Toronto

Posted about 2 months ago · $80,000 - $180,000

or apply directly on Zafin's site. We never take the application ourselves.

Is this posting real?

This role has been open
55 days
Zafin's roles stay open a median of 25 days
Reposted
No
Salary listed
Yes
34% of Zafin's roles list one
Ghost-job risk at Zafin
high
14 stale, 2 reposted of 32 open
Hiring momentum
48 roles opened in the last 90 days
↑ up vs. the prior 90 days
Last confirmed on the employer's board
2026-09-27

Measured from postings appearing on and disappearing from Zafin's own greenhouse board since 2026-08-03. Full hiring picture for Zafin.

About this role

The AI Evaluation Engineer at Zafin is responsible for ensuring AI agent solutions are accurate and production-ready within regulated banking environments. This role involves designing and executing evaluation scenarios, validating AI behavior against business and regulatory criteria, and providing actionable feedback to improve AI reliability. The engineer collaborates with various teams to ensure evaluations meet business logic and compliance standards while also leading initiatives to enhance evaluation practices and quality outcomes.

Our read on this posting2.8out of 5
benefits
4/5
freshness
1/5
career value
4/5
role clarity
5/5
pay transparency
0/5

Scored from the posting itself — how clearly the role is described, how much it says about pay and benefits, and how recently it was listed. Not a judgement of Zafin as an employer.

What you need

  • Typically 3–10+ years of relevant experience in software quality engineering, AI evaluation, AI quality engineering, machine learning evaluation, software testing, or related disciplines.
  • Experience leading evaluation activities, mentoring technical professionals, or coordinating quality initiatives is advantageous for more senior levels.
  • Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or related discipline, or equivalent practical experience.
  • Strong understanding of AI evaluation, large language model behaviour, reasoning quality, hallucination detection, safety, instruction adherence, factual accuracy, and business correctness.
  • Experience with structured software testing, regression evaluation, production-readiness assessment, and quality engineering.
  • Working knowledge of SDLC, CI/CD, automated evaluation, AI observability, and engineering delivery practices.

Nice to have

  • Experience evaluating LLMs, RAG systems, AI agents, or agentic AI platforms.
  • Experience with AI evaluation platforms such as LangSmith, OpenAI Evals, or comparable tools.
  • Experience integrating automated evaluation into CI/CD or MLOps workflows.
  • Experience with model observability, behavioural-drift detection, or AI production monitoring.
  • Banking, financial services, or other regulated industry experience.

What you get

  • Fully company-paid health benefits
  • Annual bonus opportunities
  • Generous vacation and holidays
  • RRSP matching
  • Employer-paid insurance coverage
  • Wellness resources

Worth weighing

  • No salary listed
  • Role involves working within a regulated environment which may have strict compliance requirements
  • The position is hybrid, which may require flexibility in work location

Summarised from Zafin's posting. Read the full original.

Listed by Zafin on their greenhouse job board, last confirmed open on 2026-09-27. PitchMeAI is not the employer.

More roles at Zafin

All 32 open roles at Zafin →