Legal AI Evaluator
Mercor
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
Legal AI Evaluator at Mercor
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Mercor is seeking a Legal AI Evaluator to join their team. This position is available as full-time or part-time contract work, offering a compensation of $105/hour.
Role Responsibilities
- Write and refine prompts to guide model behavior in legal use cases.
- Evaluate LLM-generated responses to legal queries for accuracy, reasoning, clarity, and completeness.
- Conduct fact-checking using trusted public sources and authoritative references.
- Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies.
- Assess tone, completeness, and appropriateness of responses for real-world legal scenarios.
- Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines.
Qualifications
Must-Have
- Minimum of 5 years of real-world professional experience in Law, supported by an associated expert degree (e.g., JD, LLB, LLM, or equivalent).
- Experience in one or more of the following sub-domains: Corporate, Business & Commercial Law; Employment, Labor & Human Rights; Public, Criminal & International Governance; Private Law.
- Significant experience using large language models (LLMs) and understanding of their use.
- Excellent writing skills and ability to clearly explain complex legal concepts.
- Comfortable evaluating complex arguments and identifying subtle reasoning or detail-level issues.
Preferred
- Prior experience with RLHF, model evaluation, or data annotation work.
- Experience writing or editing high-quality legal or analytical content.
- Experience translating legal concepts for non-legal audiences.
- Familiarity with evaluation rubrics, benchmarks, or quality scoring systems.
Application Process
The application process takes approximately 20–30 minutes to complete:
- Upload resume
- Complete an AI interview based on your resume
- Submit the form
Resources & Support
For details about the interview process and platform information, please check: talent.docs.mercor.com/welcome/welcome
For any help or support, reach out to: support@mercor.com
Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Key skills/competency
- Legal Research
- Large Language Models (LLMs)
- Prompt Engineering
- Legal Evaluation
- Fact-Checking
- Data Annotation
- Legal Writing
- Reasoning Analysis
- Compliance
- AI Ethics
How to Get Hired at Mercor
- Research Mercor's culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor. Understand their focus on connecting elite talent with AI research labs.
- Tailor your resume: Customize your resume to highlight your 5+ years of legal experience, specific sub-domain expertise, and significant experience with Large Language Models (LLMs). Use keywords like 'legal AI evaluation', 'prompt engineering', and 'data annotation'.
- Ace the AI interview: Prepare for Mercor's AI interview by practicing explaining complex legal concepts clearly and demonstrating your analytical skills in evaluating arguments and identifying nuanced issues, specifically in the context of AI-generated content.
- Showcase legal tech proficiency: Emphasize your practical experience with LLMs and any prior work in model evaluation or data annotation. Be ready to discuss how you'd apply consistent evaluation standards using taxonomies and benchmarks.
- Demonstrate strong communication: Highlight your excellent writing skills and ability to translate legal concepts for diverse audiences, aligning with the need to refine prompts and annotate responses clearly.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background