AI Model Generalist - English & Korean

About the Role: AI Model Generalist - English & Korean

Mercor, in collaboration with hackajob, is seeking exceptional tech professionals for the AI Model Generalist - English & Korean role. This position is a contract opportunity, available full-time or part-time, with geographic restrictions to South Korea and the USA. Fluency in both English and Korean is a mandatory requirement.

Mercor partners with leading AI teams to enhance the quality, usefulness, and reliability of general-purpose conversational AI systems. These systems are crucial across diverse everyday and professional scenarios, with their effectiveness hinging on clear, accurate, and helpful responses to user queries. This project specifically focuses on evaluating and improving the general chat behavior of large language models (LLMs). As an AI Model Generalist, you will assess model-generated responses across various topics, provide high-quality human feedback, and ensure AI systems communicate accurately, logically, and in line with human expectations.

What You'll Do

• Evaluate LLM-generated responses for effectiveness in answering user queries.
• Conduct thorough fact-checking using trusted public sources and external tools.
• Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies.
• Assess the reasoning quality, clarity, tone, and completeness of responses.
• Ensure model responses adhere to expected conversational behavior and system guidelines.
• Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines.

Who We're Looking For

• Hold a Bachelor’s degree.
• Native speaker or ILR 5/primary fluency (C2 on the CEFR scale) in Korean.
• Significant experience using large language models (LLMs) and understanding user interaction patterns.
• Excellent writing skills with the ability to articulate nuanced feedback clearly.
• Strong attention to detail, noticing subtle issues often overlooked by others.
• Adaptable and comfortable transitioning across diverse topics, domains, and customer requirements.
• Background or experience in domains requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering).
• Excellent college-level mathematics skills.

Nice-to-Have Specialties

• Prior experience with Reinforcement Learning from Human Feedback (RLHF), model evaluation, or data annotation work.
• Experience writing or editing high-quality written content.
• Experience comparing multiple outputs and making fine-grained qualitative judgments.
• Familiarity with evaluation rubrics, benchmarks, or quality scoring systems.

What Success Looks Like

• Identify factual inaccuracies, reasoning errors, and communication gaps in model responses.
• Produce clear, consistent, and reproducible evaluation artifacts.
• Your feedback drives measurable improvements in response quality and user experience.
• Mercor customers trust the quality of their AI systems due to issues surfaced before public release.

Why Join Mercor

At Mercor, you will operate at the forefront of human-in-the-loop AI development, directly influencing the behavior of advanced language models in real-world applications. This role offers flexible, remote contract work and a significant opportunity to contribute to AI systems utilized by millions. Contract rates are competitive, reflecting the required expertise and scope of work.

Key skills/competency

• AI Model Evaluation
• Large Language Models (LLMs)
• Korean Language Fluency
• English Language Fluency
• Data Annotation
• Fact-Checking
• Analytical Thinking
• Quality Assurance
• Human-in-the-Loop AI
• Natural Language Processing (NLP)