Engineering AI Evaluator
Mercor
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About the Role: Engineering AI Evaluator
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
This is a contract position with competitive compensation.
Role Responsibilities
- Write and refine prompts to guide model behavior in engineering scenarios.
- Evaluate LLM-generated responses for technical accuracy and applied reasoning.
- Conduct fact-checking and verify technical claims using authoritative sources.
- Annotate model responses, identifying strengths and areas for improvement.
- Assess clarity and appropriateness of explanations for different audiences.
- Ensure model responses align with expected conversational behavior and guidelines.
Qualifications
Must-Have
- PhD in Engineering or a closely related field
- Expertise in Mechanical & Physical Systems Engineering, Electrical, Electronic & Computer Engineering, Chemical, Materials & Process Engineering, or Civil, Environmental & Infrastructure Engineering
- Significant experience using large language models (LLMs)
- Excellent writing skills
- Strong attention to detail
- Experience reviewing or editing technical or academic writing
Preferred
- Experience with applied research, industry engineering workflows, or systems design
- Prior experience with RLHF, model evaluation, or data annotation work
- Experience teaching or explaining engineering concepts to non-expert audiences
- Familiarity with evaluation rubrics or structured review frameworks
Application Process
The application process takes approximately 20-30 minutes and involves uploading your resume, completing an AI interview, and submitting a form. Please complete all steps for consideration.
For more details on the interview process and platform, visit: Mercor Talent Docs. For support, contact: support@mercor.com.
Key skills/competency
- LLM Evaluation
- Prompt Engineering
- Technical Writing
- Fact-Checking
- Engineering Expertise
- Data Annotation
- Applied Research
- Attention to Detail
- AI Ethics
- RLHF
How to Get Hired at Mercor
- Research Mercor's mission: Understand their focus on connecting talent with AI research labs and their investor backing.
- Tailor your resume: Highlight PhD-level engineering, LLM experience, technical writing, and evaluation skills for the Engineering AI Evaluator role.
- Prepare for the AI interview: Practice explaining complex engineering concepts clearly and demonstrate strong communication skills for technical evaluation.
- Showcase domain expertise: Emphasize your specific engineering discipline (Mechanical, Electrical, Chemical, Civil) and any applied research or industry experience.
- Understand AI evaluation: Familiarize yourself with concepts like prompt engineering, RLHF, and data annotation as they relate to LLM performance.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background