3 days ago

Prompt Evaluator

North Star Staffing

Hybrid
Contractor
$78,000
Hybrid

Job Overview

Job TitlePrompt Evaluator
Job TypeContractor
CategoryCommerce
Experience5 Years
DegreeMaster
Offered Salary$78,000
LocationHybrid

Who's the hiring manager?

Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.

Uncover Hiring Manager

Job Description

Prompt Evaluator

One of our clients is hiring a Generalist Evaluator Expert to support a high-impact AI research initiative. This role focuses on authoring structured prompt–answer pairs and evaluation rubrics used to train and benchmark advanced language models.

About the Role

This short-term, flexible engagement is ideal for strong writers and analytical thinkers who enjoy transforming complex ideas into clear, structured instructions. You will design prompts, define grading standards, and assess AI-generated outputs within a research-driven project environment.

Key Responsibilities

  • Design detailed prompts with multiple constraints and structured instructions
  • Author high-quality “golden answers” aligned to defined expectations
  • Develop comprehensive evaluation rubrics and grading criteria
  • Run prompts through AI systems and assess outputs against standards
  • Participate in quality assurance cycles to ensure rigor and consistency
  • Contribute to benchmarking workflows before dataset integration

Ideal Qualifications

  • BS or BA from a reputable institution (completed or in progress)
  • Strong writing, editing, and critical thinking skills
  • Ability to work independently and meet structured deadlines
  • Familiarity with ChatGPT or similar AI tools for personal or professional use
  • Experience in teaching or research preferred
  • Must be based in the United States or Canada

Project Details

  • Commitment of at least 20 hours per week
  • Expected duration of approximately 2 months
  • Structured project environment with clear goals and tooling

Application Process

  • Complete a short AI-led interview (~15 minutes)
  • Complete a 45-minute written assessment focused on rubric design
  • Selected candidates will be invited to begin project work

More About the Opportunity

  • Fully remote, flexible schedule
  • Independent contractor engagement
  • Weekly payments via Stripe or Wise
  • Project scope may be extended based on performance

Key skills/competency

  • Prompt authoring
  • AI evaluation
  • Language models
  • Rubric design
  • Critical thinking
  • Strong writing
  • ChatGPT
  • Quality assurance
  • Benchmarking
  • Structured instructions

Tags:

Prompt Evaluator
AI evaluation
prompt design
rubric creation
critical thinking
quality assurance
language model training
content assessment
data benchmarking
analytical thinking
writing skills
ChatGPT
AI tools
language models
machine learning
NLP
data annotation
evaluation systems
prompt engineering
text generation
AI research

Share Job:

How to Get Hired at North Star Staffing

  • Research North Star Staffing's mission: Understand their client focus in AI research and how the Prompt Evaluator role contributes.
  • Customize your resume: Tailor your resume to highlight strong writing, critical thinking, and any experience with AI tools like ChatGPT for the Prompt Evaluator position.
  • Prepare for AI-led interview: Practice articulating your analytical process and experience in a structured, concise manner for North Star Staffing's initial AI interview.
  • Master the written assessment: Demonstrate your ability to design detailed prompts, author quality answers, and develop clear evaluation rubrics during the 45-minute written test.
  • Showcase independent work: Emphasize your ability to work autonomously, meet structured deadlines, and manage project commitments effectively, as North Star Staffing values independent contractors.

Frequently Asked Questions

Find answers to common questions about this job opportunity

Explore similar opportunities that match your background