AI Evaluation Analyst

Role Overview

The AI Evaluation Analyst is responsible for assessing the quality, performance, and reliability of AI systems and model outputs. This role supports the development and continuous improvement of AI products by analyzing results, identifying patterns, and providing structured feedback to technical and cross-functional teams.

Key Responsibilities

• Review and evaluate AI generated outputs for accuracy, relevance, safety, and consistency
• Apply predefined evaluation frameworks, rubrics, and quality standards
• Conduct qualitative and quantitative analysis on model performance
• Identify edge cases, failure patterns, and areas for improvement
• Document findings clearly and provide actionable recommendations
• Collaborate with data scientists, product teams, and engineers to refine models
• Support testing cycles, experiments, and benchmarking initiatives
• Maintain high attention to detail when handling sensitive or confidential data

Skills and Qualifications

• Strong analytical and critical thinking skills
• Ability to interpret guidelines and apply structured evaluation criteria
• Excellent written communication and documentation abilities
• Comfortable working with datasets, spreadsheets, and reporting tools
• Familiarity with AI concepts such as machine learning, NLP, or large language models is a plus
• Detail oriented, consistent, and able to manage repetitive review tasks
• Ability to work independently while meeting quality and turnaround expectations

Preferred Background

• Experience in quality assurance, data analysis, research, or content review
• Exposure to AI products, prompt evaluation, model testing, or annotation workflows
• Comfort working in fast evolving, experimentation driven environments

Key skills/competency

• AI evaluation
• Qualitative analysis
• Quantitative analysis
• Model performance
• Feedback loops
• Edge case identification
• Data analysis
• NLP
• LLM
• Quality assurance