AI Conversation Evaluator

Job Overview

Mercor is seeking highly skilled Generalists (Bachelor’s degree minimum) to support a fast-moving project with one of the world’s leading AI labs. This initiative focuses on improving AI systems designed for therapeutic-style conversations by evaluating, comparing, and refining model-generated responses.

Role Responsibilities

In this role, you will review pairs of AI-generated replies and assess them against structured criteria (e.g., empathy, emotional insight, communication quality, and appropriateness for supportive dialogue). When needed, you will also produce clear, improved rewrites that better meet the project’s standards. Your work will directly support research teams as they strengthen safe, supportive, and human-aligned AI behavior in mental-health-oriented conversation settings.

Important Notes

This project has an accelerated timeline and requires contributors who can begin immediately following a mandatory onboarding session.

We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.

Contract and Payment Terms

• You will be engaged as an independent contractor.
• This is a fully remote role that can be completed on your own schedule.
• Projects can be extended, shortened, or concluded early depending on needs and performance.
• Your work at Mercor will not involve access to confidential or proprietary information from any employer, client, or institution.
• Payments are weekly on Stripe or Wise based on services rendered.
• Please note: We are unable to support H1-B or STEM OPT candidates at this time.

Key skills/competency

• AI evaluation
• Natural Language Processing
• Empathy assessment
• Conversational AI
• Content moderation
• Linguistic analysis
• Feedback provision
• Model refinement
• Communication quality
• Therapeutic dialogue