AI Model Generalist - English & Brazilian Portuguese

About the Role: AI Model Generalist - English & Brazilian Portuguese

hackajob is collaborating with Mercor to connect exceptional tech professionals for this vital role. Mercor partners with leading AI teams to enhance the quality, usefulness, and reliability of general-purpose conversational AI systems. These systems are integral to diverse everyday and professional scenarios, with their effectiveness directly linked to clear, accurate, and helpful responses to user queries. This project specifically focuses on evaluating and improving general chat behavior in large language models (LLMs). As an AI Model Generalist, you will be instrumental in assessing model-generated responses across various topics, providing high-quality human feedback, and ensuring AI systems communicate in an accurate, well-reasoned manner, aligned with human expectations.

This is a flexible, remote contract opportunity, available as full-time or part-time work, restricted to candidates located in Brazil or the USA.

What You’ll Do

• Evaluate LLM-generated responses based on their ability to effectively answer user queries.
• Conduct thorough fact-checking utilizing trusted public sources and external tools.
• Generate high-quality human evaluation data by annotating response strengths, identifying areas for improvement, and pinpointing factual inaccuracies.
• Assess the reasoning quality, clarity, tone, and completeness of AI-generated responses.
• Ensure model responses consistently align with expected conversational behavior and system guidelines.
• Apply consistent annotations by diligently following clear taxonomies, benchmarks, and detailed evaluation guidelines.

Who You Are

• You possess a Bachelor’s degree.
• You are a native speaker or have ILR 5/primary fluency (C2 on the CEFR scale) in Brazilian Portuguese.
• You have significant experience using large language models (LLMs) and a strong understanding of their applications.
• You demonstrate excellent writing skills, capable of clearly articulating nuanced feedback.
• You exhibit strong attention to detail, consistently noticing subtle issues often overlooked by others.
• You are adaptable and comfortable transitioning between diverse topics, domains, and customer requirements.
• You have a background or experience in fields requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering).
• You possess excellent college-level mathematics skills.

Nice-to-Have Specialties

• Prior experience with RLHF, model evaluation, or data annotation work.
• Experience in writing or editing high-quality written content.
• Experience comparing multiple outputs and making fine-grained qualitative judgments.
• Familiarity with evaluation rubrics, benchmarks, or quality scoring systems.

What Success Looks Like

• Consistently identifying factual inaccuracies, reasoning errors, and communication gaps in model responses.
• Producing clear, consistent, and reproducible evaluation artifacts.
• Your feedback directly contributing to measurable improvements in response quality and user experience.
• Mercor's customers trusting the quality of their AI systems because your evaluations surface issues before public release.

Key skills/competency

• LLM Evaluation
• Brazilian Portuguese Fluency
• English Fluency
• Fact-Checking
• Data Annotation
• Analytical Thinking
• Writing Skills
• Attention to Detail
• AI Systems
• Natural Language Processing