AI Language Model Evaluator
@ Call For Referral

Hybrid
$75,000
Hybrid
Part Time
Posted 16 hours ago

Your Application Journey

Personalized Resume
Apply
Email Hiring Manager
Interview

Email Hiring Manager

XXXXXXXXX XXXXXXXXXXX XXXXXXXXX****** @callforreferral.com
Recommended after applying

Job Details

About the Role

The AI Language Model Evaluator role is a part-time, remote position for writing experts. This role involves designing and optimizing prompts, developing evaluation standards, conducting model testing and grading, and supporting benchmarking and quality assurance processes for a high-impact AI research project.

Key Responsibilities

  • Create detailed prompts with multiple constraints and instructions.
  • Establish high-level expectations and develop evaluation rubrics.
  • Test language models and assess outputs against set standards.
  • Collaborate in QA review processes to ensure consistency and reliability.

Minimum Qualifications

  • BS or BA (completed or in progress) from a reputable institution.
  • Strong writing and critical thinking skills.
  • Ability to work independently and meet deadlines.
  • Familiarity with ChatGPT or similar tools.
  • Based in US or Canada.

Preferred Qualifications

  • Experience in teaching or research.

Application & Onboarding Process

Complete an AI-led interview (~15 minutes) and a 45-minute written assessment. If selected, you will join a structured project environment with clear goals, working at least 20 hours per week for about one month.

Key Skills/Competency

  • Prompt Design
  • Evaluation Rubrics
  • Model Testing
  • Quality Assurance
  • Writing Skills
  • Critical Thinking
  • Instructional Clarity
  • Remote Work
  • Research
  • Analytical Skills
Apply without a personalized resume

How to Get Hired at Call For Referral

🎯 Tips for Getting Hired

  • Customize Your Resume: Tailor skills to language model evaluation requirements.
  • Highlight Experience: Emphasize writing and critical thinking competencies.
  • Prepare Examples: Showcase prompt design and evaluation tasks.
  • Research Call For Referral: Understand company mission and project goals.

📝 Interview Preparation Advice

Technical Preparation

Review prompt design fundamentals.
Practice language model testing tasks.
Study rubric creation techniques.
Understand evaluation metrics.

Behavioral Questions

Describe a challenging remote work scenario.
Explain managing tight deadlines independently.
Discuss adjustment to feedback changes.
Share teamwork experiences remotely.

Frequently Asked Questions