Generalist Evaluator Expert
Talent Wall
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About the Role
Mercor is seeking detail-oriented writing experts to contribute to a high-impact AI research project with a leading lab. Freelancers will author prompt–golden answer pairs that train and evaluate advanced language models. This is a short-term, flexible opportunity for professionals with strong academic backgrounds and a knack for instructional clarity. It is ideal for those who enjoy distilling complex concepts into well-crafted text.
Job Details
- Design and Optimize Prompts: Create detailed prompts with multiple constraints and instructions.
- Define and Document Evaluation Standards: Establish high-level expectations for correct responses in general consumer contexts, and develop comprehensive rubric.
- Conduct Model Testing and Grading: Run prompts through models and assess preliminary outputs against expectations.
- Support Benchmarking and Quality Assurance: Collaborate in QA review processes to ensure prompt tasks and rubrics meet rigor, maintaining consistency and reliability before integration into official benchmarks.
What You'll Bring
Minimum Qualifications:
- BS or BA from a reputable institution completed or in progress
- Strong writing and critical thinking skills.
- Ability to work independently and meet deadlines.
- Significant familiarity with ChatGPT or similar tools for personal decision-making or hobbies / general interests.
- US or Canada based.
Preferred Qualifications:
- Experience in teaching or research.
Application Process
- Complete an AI-led interview, this should take around 15 minutes.
- Complete a 45-minute written assessment that will guide you through writing rubrics.
- If selected, you will be invited to work on the project.
Project Details
Expect to contribute at least 20 hours per week.
Expect a commitment of around 2 months.
You’ll be working in a structured project environment with clear goals and tools.
We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.
Contract and Payment Terms
You will be engaged as an independent contractor.
This is a fully remote role that can be completed on your own schedule.
Projects can be extended, shortened, or concluded early depending on needs and performance.
Your work at Mercor will not involve access to confidential or proprietary information from any employer, client, or institution.
Payments are weekly on Stripe or Wise based on services rendered.
Please note: We are unable to support H1-B or STEM OPT candidates at this time.
Key skills/competency
- AI Research
- Prompt Engineering
- Language Model Evaluation
- Technical Writing
- Critical Thinking
- Rubric Development
- Quality Assurance
- Instructional Design
- Data Annotation
- Project Management
How to Get Hired at Talent Wall
- Research Talent Wall's mission: Study their platform, values, and how they connect experts with leading AI projects like Mercor's.
- Highlight AI and writing expertise: Tailor your resume to showcase strong writing, critical thinking, and experience with generative AI tools.
- Prepare for skills assessments: Practice crafting detailed prompts and developing clear evaluation rubrics for the written exercise.
- Demonstrate independent work ethic: Emphasize your ability to work autonomously, meet deadlines, and thrive in a remote, structured project environment.
- Showcase academic background: Leverage any teaching, research, or strong academic achievements relevant to clear communication and analysis.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background