Professional Evaluator - Fully Remote | Upto $35/hr Hourly
Mercor · New York, NY
- Hybrid
- Part-time
- $35,000 / year
- New York, NY
Job highlights
- Evaluate AI responses for accuracy and usefulness.
- Apply professional domain expertise in evaluation.
- Identify errors and provide evidence-based justifications.
- Work 20 hours/week on a contract basis.
- Join a leading AI talent platform.
About the role
About The Job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: AI Model Evaluation Contractor
Type: Contract
Compensation: $25–$35/hour
Commitment: 20 hours/week
Role Responsibilities
- Write realistic prompts reflecting professional and consumer domain-specific guidance.
- Evaluate AI-generated responses for factual accuracy, regulatory correctness, and practical usefulness.
- Identify fabricated claims, incorrect references, or misleading reasoning in model outputs.
- Score and rank multiple model responses using structured rubrics across dimensions.
- Provide written justifications with specific evidence for each evaluation.
Qualifications
Must-Have
- Professional experience applying domain expertise in a practitioner or advisory capacity.
- Familiarity with industry-specific standards, regulations, or clinical guidelines.
- Strong written communication and critical reasoning skills.
Application Process (Takes 20–30 mins to complete)
- Submit your resume to begin.
- Complete the Model Response Evaluation assessment.
Resources & Support
For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome/welcome
For any help or support, reach out to: support@mercor.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Key skills/competency
- AI Model Evaluation Contractor
- Domain Expertise
- Critical Reasoning
- Written Communication
- Factual Accuracy
- Regulatory Correctness
- Practical Usefulness
- Response Evaluation
- Structured Rubrics
- Professional Experience
Skills & topics
- AI Model Evaluation
- Contractor
- Remote Work
- Domain Expertise
- Critical Reasoning
- AI Research
- Mercor
- Technical Talent
- Response Evaluation
- Data Annotation
How to get hired
- Tailor your resume: Highlight your domain expertise and critical reasoning skills relevant to AI model evaluation.
- Ace the assessment: Practice evaluating responses for accuracy and correctness to excel in the Model Response Evaluation.
- Review resources: Understand Mercor's platform and interview process via their documentation.
- Apply promptly: Complete your resume submission and AI interview steps swiftly for daily review.
- Seek support: Contact support@mercor.com for any assistance during the application process.
Technical preparation
Understand AI response evaluation criteria.,Practice identifying factual inaccuracies.,Familiarize with regulatory standards.,Develop structured reasoning for feedback.
Behavioral questions
Describe a time you identified misinformation.,How do you ensure objectivity in analysis?,Showcase your critical thinking skills.,Explain your domain expertise application.
Frequently asked questions
- What is the primary responsibility of an AI Model Evaluation Contractor at Mercor?
- As an AI Model Evaluation Contractor at Mercor, your main role is to evaluate AI-generated responses for factual accuracy, regulatory correctness, and practical usefulness, ensuring they meet professional and consumer domain-specific standards.
- What kind of professional experience is required for the AI Model Evaluation Contractor role?
- Mercor requires professional experience where you've applied your domain expertise in a practitioner or advisory capacity. Familiarity with industry-specific standards, regulations, or clinical guidelines is also essential.
- How much time does the application process for the AI Model Evaluation Contractor role take?
- The application process for the AI Model Evaluation Contractor role is designed to be efficient, typically taking between 20 to 30 minutes to complete, including resume submission and an assessment.
- What is the compensation for an AI Model Evaluation Contractor at Mercor?
- The AI Model Evaluation Contractor role offers a compensation of $25 to $35 per hour, based on your qualifications and experience.
- Is the AI Model Evaluation Contractor position a full-time role?
- No, the AI Model Evaluation Contractor position is a contract role requiring a commitment of approximately 20 hours per week.
- How does Mercor ensure the quality of AI model evaluations?
- Mercor ensures quality by having professionals with deep domain expertise evaluate AI responses against structured rubrics and provide evidence-based justifications for their assessments.
- Who are Mercor's notable investors, and what does this mean for the AI Model Evaluation Contractor role?
- Mercor is backed by prominent investors such as Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey. This strong backing indicates a robust and well-supported platform, offering stability and potential growth opportunities for contractors.
- What resources are available if I need help with the application or platform for the AI Model Evaluation Contractor role?
- You can find detailed information about the interview process and platform on Mercor's talent documentation site: https://talent.docs.mercor.com/welcome/welcome. For direct support, you can reach out to support@mercor.com.