Fact Checker
Mercor · Australia
This listing has closed — view similar roles below.
- Hybrid
- Part-time
- $45,000 / year
- Australia
Job highlights
- Evaluate AI search responses for accuracy and quality.
- Assess search query effectiveness and model usage.
- Compare model outputs and provide rationales.
- Write and refine evaluation criteria and prompts.
- Work independently in an asynchronous remote setting.
About the role
About The Job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Search Generalist Expert
Type: Contract
Compensation: $10–$30/hour
Location: Remote
Role Responsibilities
- Evaluate AI-generated search responses for factual accuracy, helpfulness, clarity, completeness, and overall quality.
- Assess whether models use search appropriately and whether search queries are well-formed and effective.
- Compare model responses side by side and provide concise, defensible rationales.
- Write and refine prompts, golden answers, rubric criteria, and edge cases for search-related evaluations.
- Apply project guidelines consistently across ambiguous, multi-step, and real-world search tasks.
- Identify recurring failure modes and escalate unclear cases or rubric gaps to project leads.
Qualifications
Must-Have
- Excellent written English and strong online research skills.
- Strong judgment when synthesizing information from multiple sources.
- Ability to distinguish factual accuracy from fluency, confidence, or style.
- High attention to detail and comfort following structured guidelines.
- Reliable, self-directed, and responsive in an asynchronous remote environment.
Preferred
- Experience in search quality, fact-checking, content evaluation, trust and safety, annotation, QA, or prompt/rubric writing.
- Familiarity with search evaluation concepts such as factuality, helpfulness, severity, side-by-side comparisons, or tool-use assessment.
- Experience working with LLM evaluation workflows or human data projects.
- Multilingual skills are a plus.
- Bachelor’s degree preferred; advanced degree or strong professional background is a plus.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
For any help or support, reach out to: support@mercor.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Key skills/competency
- Fact Checking
- Research Skills
- Content Evaluation
- Quality Assurance
- Prompt Writing
- LLM Evaluation
- Attention to Detail
- Written English
- Information Synthesis
- Remote Work
Skills & topics
- Fact Checker
- AI Evaluation
- Search Quality
- Content Review
- Remote Work
- Contract Role
- Research
- Quality Assurance
- LLM
- Prompt Engineering
How to get hired
- Tailor your resume: Highlight your research, fact-checking, and attention to detail skills.
- Prepare for AI interview: Be ready to discuss your resume and qualifications.
- Follow instructions carefully: Ensure all application steps are completed accurately.
- Demonstrate remote readiness: Emphasize your self-direction and communication skills.
- Understand Mercor's mission: Show how your skills support connecting talent with AI labs.
Technical preparation
Practice online research techniques.,Develop critical evaluation skills.,Familiarize with AI search concepts.,Review prompt writing principles.
Behavioral questions
How do you ensure accuracy in research?,Describe handling ambiguous guidelines.,How do you work independently remotely?,How do you synthesize information from multiple sources?
Frequently asked questions
- What is the application process for the Fact Checker role at Mercor?
- The application process for the Fact Checker position at Mercor involves uploading your resume, completing an AI-driven interview based on your resume, and submitting the form. The entire process typically takes 20-30 minutes. Applications are reviewed daily.
- What are the key responsibilities of a Fact Checker at Mercor?
- As a Fact Checker at Mercor, your primary responsibilities include evaluating the factual accuracy, helpfulness, clarity, and completeness of AI-generated search responses. You will also assess search query effectiveness, compare model responses, write prompts and evaluation criteria, and identify failure modes. This role requires applying project guidelines consistently to real-world search tasks.
- What qualifications are essential for the Fact Checker role?
- Essential qualifications for the Fact Checker role include excellent written English, strong online research skills, sound judgment in synthesizing information, and the ability to distinguish factual accuracy from fluency. A high attention to detail and comfort with structured guidelines, along with being self-directed and responsive in a remote environment, are also crucial.
- What experience is preferred for this Fact Checker position at Mercor?
- Preferred experience for this Fact Checker role at Mercor includes working in search quality, fact-checking, content evaluation, trust and safety, annotation, QA, or prompt/rubric writing. Familiarity with search evaluation concepts and experience with LLM evaluation workflows or human data projects are also beneficial. Multilingual skills and a relevant degree are a plus.
- Is the Fact Checker position at Mercor remote?
- Yes, the Fact Checker position at Mercor is a remote role. This means you can work from any location, and it requires you to be reliable, self-directed, and responsive in an asynchronous remote environment.
- What is the compensation for the Fact Checker role at Mercor?
- The compensation for the Fact Checker role at Mercor is hourly, ranging from $10 to $30 per hour. The specific rate may depend on your qualifications and experience.
- What kind of company is Mercor and what do they do?
- Mercor is a company that connects elite creative and technical talent with leading AI research labs. They are backed by prominent investors and are focused on the AI and talent matching industries.