Mathematics AI Evaluator
Mercor
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About The Job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Mathematics AI Evaluator
This role is available as full-time or part-time contract work with competitive hourly compensation.
Role Responsibilities
- Write and refine prompts to guide model behavior in mathematical contexts.
- Evaluate LLM-generated responses to mathematics-related queries for correctness, rigor, and logical coherence.
- Verify mathematical claims, derivations, and proofs using domain expertise.
- Conduct fact-checking using authoritative public sources and domain knowledge.
- Annotate model responses by identifying strengths, areas of improvement, and inaccuracies.
- Ensure model responses align with expected conversational behavior and system guidelines.
Qualifications
Must-Have:
- PhD in Mathematics or a closely related field.
- Demonstrated experience in Probability & Statistics.
- Significant experience using large language models (LLMs).
- Excellent writing skills for explaining complex mathematical concepts.
- Strong attention to detail to notice subtle issues.
- Experience reviewing or editing technical or academic writing.
Preferred:
- Prior experience with RLHF, model evaluation, or data annotation work.
- Experience teaching or mentoring mathematical concepts to non-experts.
- Familiarity with evaluation rubrics, benchmarks, or structured review frameworks.
Application Process
Upload resume, complete an AI interview based on your resume, and submit a form. The process takes 20–30 minutes.
Resources & Support
For interview process details, visit this link. For help or support, email support@mercor.com.
Key skills/competency
- Mathematics
- LLM Evaluation
- Probability
- Statistics
- Prompt Engineering
- Proof Verification
- Fact-checking
- Technical Writing
- Attention to Detail
- AI Models
How to Get Hired at Mercor
- Research Mercor's culture: Understand mission, investor background, and AI focus.
- Customize your resume: Highlight PhD and math expertise.
- Prepare examples: Showcase prompt writing and evaluation skills.
- Practice interviews: Focus on technical and academic experiences.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background