Legal AI Evaluator

About Mercor

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position Overview: Legal AI Evaluator

This role is available as full-time or part-time contract work with a compensation of $105 per hour. The position involves evaluating and refining AI-generated legal content.

Role Responsibilities

• Write and refine prompts to guide model behavior in legal use cases.
• Evaluate LLM-generated legal responses for accuracy, reasoning, clarity, and completeness.
• Conduct fact-checking using trusted public sources.
• Annotate responses by identifying strengths and areas for improvement.
• Assess tone, completeness, and appropriateness for real-world legal scenarios.
• Apply consistent evaluation standards using clear taxonomies, benchmarks, and guidelines.

Qualifications

Must-Have: Minimum 5 years professional legal experience with a JD, LLB, LLM or equivalent. Experience in Corporate, Business & Commercial Law; Employment, Labor & Human Rights; Public, Criminal & International Governance; or Private Law. Experience with LLMs and excellent writing skills are required.

Preferred: Experience with RLHF, model evaluation, data annotation, writing or editing legal content, and translating legal concepts for non-legal audiences. Familiarity with evaluation rubrics and quality scoring systems is a plus.

Application Process & Resources

Complete the following steps to apply:

• Upload your resume
• Complete an AI interview based on your resume
• Submit the application form

For more information, visit: Interview Process & Platform Info or contact support at support@mercor.com.

Key skills/competency

• Legal Evaluation
• LLM Assessment
• Prompt Engineering
• Fact-Checking
• Legal Writing
• Taxonomy Application
• Data Annotation
• Analytical Reasoning
• Model Evaluation
• Quality Scoring