Engineering AI Evaluator
Mercor
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About Mercor
Mercor specializes in connecting top-tier creative and technical talent with leading AI research labs. Headquartered in San Francisco, Mercor boasts a prestigious list of investors including Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
The Opportunity: Engineering AI Evaluator
Mercor is seeking an Engineering AI Evaluator for a contract position. This remote role offers a compensation of $73 per hour.
Role Responsibilities
- Write and refine prompts to guide model behavior in engineering scenarios.
- Evaluate LLM-generated responses for technical accuracy and applied reasoning.
- Conduct fact-checking and verify technical claims using authoritative sources.
- Annotate model responses, identifying strengths and areas for improvement.
- Assess clarity and appropriateness of explanations for different audiences.
- Ensure model responses align with expected conversational behavior and guidelines.
Qualifications
Must-Have
- PhD in Engineering or a closely related field.
- Expertise in Mechanical & Physical Systems Engineering, Electrical, Electronic & Computer Engineering, Chemical, Materials & Process Engineering, or Civil, Environmental & Infrastructure Engineering.
- Significant experience using large language models (LLMs).
- Excellent writing skills.
- Strong attention to detail.
- Experience reviewing or editing technical or academic writing.
Preferred
- Experience with applied research, industry engineering workflows, or systems design.
- Prior experience with RLHF, model evaluation, or data annotation work.
- Experience teaching or explaining engineering concepts to non-expert audiences.
- Familiarity with evaluation rubrics or structured review frameworks.
Application Process
The application process takes approximately 20-30 minutes and involves:
- Uploading your resume.
- Completing an AI interview based on your resume.
- Submit the application form.
Mercor's team reviews applications daily. Please ensure all AI interview and application steps are completed for consideration.
Resources & Support
For details on the interview process and platform information, visit: Mercor Talent Docs. For any assistance, contact: support@mercor.com.
Key skills/competency
- LLM Evaluation
- Prompt Engineering
- Technical Writing
- Engineering Expertise
- Fact-Checking
- Data Annotation
- Applied Research
- Quality Assurance
- AI Ethics
- Technical Editing
How to Get Hired at Mercor
- Research Mercor's mission: Understand their focus on connecting talent with leading AI labs and their investor backing to tailor your application.
- Customize your resume: Highlight your PhD, engineering specialization, and extensive LLM experience for the Engineering AI Evaluator role.
- Prepare for AI interviews: Practice articulating your technical evaluation skills and prompt engineering experience effectively for Mercor's AI interview.
- Showcase writing and detail orientation: Emphasize your strong technical writing, editing, and fact-checking abilities crucial for this Mercor position.
- Demonstrate domain expertise: Clearly illustrate your deep knowledge in relevant engineering fields and experience explaining complex concepts.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background