Multimodal GenAI Evaluation Analyst

Position Overview

iMerit seeks detail-oriented and analytically minded Multimodal GenAI Evaluation Analysts to perform highly nuanced evaluations of AI system outputs across different modalities: text, image, video, and multimodal interactions. Analysts will assess the accuracy, appropriateness, quality, clarity, and cultural alignment of model outputs against complex guidelines, ensuring that results align with project standards and real-world use cases. These evaluations will directly inform the development and fine-tuning of advanced large language models (LLMs), vision models (LVMs), and multimodal AI systems.

Role Responsibilities

• Evaluate outputs generated by LLMs across multiple modalities (text, image captions, video descriptions, and multimodal prompts).
• Assess quality against project-specific criteria such as correctness, coherence, completeness, style, cultural appropriateness, and safety.
• Identify subtle errors, hallucinations, or biases in AI responses.
• Apply domain expertise and logical reasoning to resolve ambiguous or unclear outputs.
• Provide detailed written feedback, tagging, and scoring of outputs to ensure consistency across the evaluation team.
• Escalate unclear cases and contribute to refining evaluation guidelines.
• Collaborate with Project Managers and Quality Leads to meet accuracy, reliability, and turnaround benchmarks.

Skills & Competencies

• Strong critical reading, observational, and evaluative skills across different modalities.
• Ability to articulate nuanced judgments with precision and clarity.
• Excellent English comprehension (CEFR B2 or above); additional languages a plus.
• Familiarity with LLMs, generative AI, and multimodal systems.
• Strong attention to detail and ability to apply guidelines consistently.
• Awareness of cultural and linguistic nuances, including potential bias and harm in AI outputs.
• Comfort with evolving workflows, rapid feedback cycles, and complex quality frameworks.

Requirements

• Bachelor's degree/ diploma or equivalent educational qualification.
• 1+ years of experience in data annotation, LLM evaluation, content moderation, or related AI/ML domains.
• Demonstrated experience working with data annotation tools and software platforms.
• Strong understanding of language and multimodal communication (instruction following in image generation, fact-checking, narrative coherence in video, etc.).
• Ability to adapt quickly to changing project directions and fast-paced work environments.
• Previous experience creating or annotating complex data specifically for Large Language Model (LLM) training.
• Prior exposure to generative AI, prompt engineering, or LLM fine-tuning workflows is a plus.

While moderation of high-harm/high-risk material is not part of this role, candidates should be aware that occasional exposure to NSFW or otherwise sensitive content may occur due to imperfections in client-provided datasets. Applicants should indicate that they are comfortable working in environments where such incidental exposure is a possibility.

What We Offer

• Opportunities to shape the evaluation standards for next-generation multimodal AI systems.
• Innovative and supportive global working environment.
• Competitive compensation and flexible remote working arrangements.
• Continuous learning and growth in applied AI evaluation.

Selection Process

Please acknowledge that you agree to the selection process below:

• You will receive an iMerit platform assessment (15–30 minutes). If successfully completed, you’ll be invited to join the first project.
• After onboarding, once you’ve completed 10 hours of work, a quality test will be conducted.
• If you pass the quality test, you’ll continue on a 3-month project and will be invited to participate in upcoming projects.

Note:

• You will complete a quick 15–30 minute assessment. This requires downloading a browser extension, which can be removed once the assessment is completed.
• ID verification and background check are required.
• Onboarding will be completed through iMerit’s platform.

For Digital Nomads: If you are currently traveling, please let us know. This ensures any discrepancies between your current location and your work authorization location do not affect your application.

Commitment

• Minimum 20 hours per week (flexible schedule).
• You may work more hours if desired.

Hourly Rates

• Malaysia – $5/hr
• Mexico, Colombia, Brazil, Costa Rica – $8.50/hr
• Argentina, Poland, Bulgaria, Romania, Malta, Latvia, Lithuania, UAE – $13/hr
• Portugal, Italy, Greece, Spain – $15.50/hr
• Canada, Australia, New Zealand, United Kingdom, Ireland, US, Finland, France, Sweden, Belgium, Austria, Denmark, Germany, Luxembourg, Estonia – $22/hr

Key skills/competency

• AI Evaluation
• Large Language Models (LLMs)
• Multimodal AI Systems
• Data Annotation
• Content Moderation
• Critical Thinking
• Linguistic Nuance
• Generative AI
• Quality Assurance
• Prompt Engineering