Legal AI Evaluator
Mercor
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About Mercor
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Legal AI Evaluator
This role can be full-time or part-time contract work. Compensation is $105/hour. The role is open to candidates located in USA, EU, UK, and Canada.
Role Responsibilities
- Write and refine prompts to guide model behavior for legal use cases.
- Evaluate LLM-generated responses to legal queries for accuracy and clarity.
- Conduct fact-checking using trusted public sources and authoritative references.
- Annotate responses by identifying strengths and areas for improvement.
- Assess tone, completeness, and appropriateness for real-world legal scenarios.
- Apply consistent evaluation standards following clear taxonomies, benchmarks, and guidelines.
Qualifications
Must-Have: Minimum 5 years of professional legal experience with an expert degree (JD, LLB, LLM or equivalent). Proven experience in areas including Corporate, Business & Commercial Law; Employment, Labor & Human Rights; Public, Criminal & International Governance; Private Law. Demonstrated expertise using large language models and excellent writing skills with clear explanation of complex legal concepts. Ability to evaluate complex arguments and subtle details.
Preferred: Experience with RLHF, model evaluation, or data annotation; writing or editing high-quality legal or analytical content; translating legal concepts for non-legal audiences; familiarity with evaluation rubrics, benchmarks, or quality scoring systems.
Application Process
The application process takes 20–30 minutes and involves uploading your resume, completing an AI interview based on your resume, and submitting a form.
Resources & Support
For interview process details, please check: https://talent.docs.mercor.com/welcome/welcome. For support, reach out via email at support@mercor.com.
Key skills/competency
- Legal Analysis
- LLM Evaluation
- Prompt Engineering
- Fact-Checking
- Legal Writing
- Data Annotation
- Evaluation Standards
- Guideline Adherence
- Analytical Reasoning
- Communication
How to Get Hired at Mercor
- Research Mercor's culture: Study their mission, investors, and processes.
- Customize your resume: Highlight relevant legal and AI skills.
- Prepare detailed examples: Demonstrate past legal evaluations and writing.
- Practice interview insights: Get ready for AI-based questions.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background