or apply directly on Zafin's site. We never take the application ourselves.
Is this posting real?
- This role has been open
- 55 days Zafin's roles stay open a median of 25 days
- Reposted
- No
- Salary listed
- Yes 34% of Zafin's roles list one
- Ghost-job risk at Zafin
- high 14 stale, 2 reposted of 32 open
- Hiring momentum
- 48 roles opened in the last 90 days ↑ up vs. the prior 90 days
- Last confirmed on the employer's board
- 2026-09-27
Measured from postings appearing on and disappearing from Zafin's own greenhouse board since 2026-08-03. Full hiring picture for Zafin.
About this role
The AI Evaluation Engineer at Zafin is responsible for ensuring AI agent solutions are accurate and production-ready within regulated banking environments. This role involves designing and executing evaluation scenarios, validating AI behavior against business and regulatory criteria, and providing actionable feedback to improve AI reliability. The engineer collaborates with various teams to ensure evaluations meet business logic and compliance standards while also leading initiatives to enhance evaluation practices and quality outcomes.
- benefits
- 4/5
- freshness
- 1/5
- career value
- 4/5
- role clarity
- 5/5
- pay transparency
- 0/5
Scored from the posting itself — how clearly the role is described, how much it says about pay and benefits, and how recently it was listed. Not a judgement of Zafin as an employer.
What you need
- Typically 3–10+ years of relevant experience in software quality engineering, AI evaluation, AI quality engineering, machine learning evaluation, software testing, or related disciplines.
- Experience leading evaluation activities, mentoring technical professionals, or coordinating quality initiatives is advantageous for more senior levels.
- Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or related discipline, or equivalent practical experience.
- Strong understanding of AI evaluation, large language model behaviour, reasoning quality, hallucination detection, safety, instruction adherence, factual accuracy, and business correctness.
- Experience with structured software testing, regression evaluation, production-readiness assessment, and quality engineering.
- Working knowledge of SDLC, CI/CD, automated evaluation, AI observability, and engineering delivery practices.
Nice to have
- Experience evaluating LLMs, RAG systems, AI agents, or agentic AI platforms.
- Experience with AI evaluation platforms such as LangSmith, OpenAI Evals, or comparable tools.
- Experience integrating automated evaluation into CI/CD or MLOps workflows.
- Experience with model observability, behavioural-drift detection, or AI production monitoring.
- Banking, financial services, or other regulated industry experience.
What you get
- Fully company-paid health benefits
- Annual bonus opportunities
- Generous vacation and holidays
- RRSP matching
- Employer-paid insurance coverage
- Wellness resources
Worth weighing
- No salary listed
- Role involves working within a regulated environment which may have strict compliance requirements
- The position is hybrid, which may require flexibility in work location
Summarised from Zafin's posting. Read the full original.
Listed by Zafin on their greenhouse job board, last confirmed open on 2026-09-27. PitchMeAI is not the employer.
More roles at Zafin
- AI Knowledge & Governance LeadToronto
- Delivery ExecutiveToronto
- AI Operations LeaderToronto
- Agent Configuration SpecialistToronto
- Principal ArchitectToronto
- AI Mobile Engineer- AndroidToronto
- Lead Quality EngineerToronto
- Technical Product ManagerToronto