18 hours ago

Prompt Evaluator

North Star Staffing

Hybrid
Contractor
$78,000
Hybrid

Job Overview

Job TitlePrompt Evaluator
Job TypeContractor
CategoryCommerce
Experience5 Years
DegreeMaster
Offered Salary$78,000
LocationHybrid

Who's the hiring manager?

Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.

Uncover Hiring Manager

Job Description

Prompt Evaluator at North Star Staffing

North Star Staffing is seeking a Generalist Evaluator Expert to support a high-impact AI research initiative. This remote, hourly contract role focuses on authoring structured prompt–answer pairs and evaluation rubrics used to train and benchmark advanced language models.

About the Role

This short-term, flexible engagement is ideal for strong writers and analytical thinkers who enjoy transforming complex ideas into clear, structured instructions. You will design prompts, define grading standards, and assess AI-generated outputs within a research-driven project environment.

Key Responsibilities

  • Design detailed prompts with multiple constraints and structured instructions.
  • Author high-quality “golden answers” aligned to defined expectations.
  • Develop comprehensive evaluation rubrics and grading criteria.
  • Run prompts through AI systems and assess outputs against standards.
  • Participate in quality assurance cycles to ensure rigor and consistency.
  • Contribute to benchmarking workflows before dataset integration.

Ideal Qualifications

  • BS or BA from a reputable institution (completed or in progress).
  • Strong writing, editing, and critical thinking skills.
  • Ability to work independently and meet structured deadlines.
  • Familiarity with ChatGPT or similar AI tools for personal or professional use.
  • Experience in teaching or research preferred.
  • Must be based in the United States or Canada.

Project Details

  • Commitment of at least 20 hours per week.
  • Expected duration of approximately 2 months.
  • Structured project environment with clear goals and tooling.

More About the Opportunity

  • Fully remote, flexible schedule.
  • Independent contractor engagement.
  • Weekly payments via Stripe or Wise.
  • Project scope may be extended based on performance.

Key skills/competency

  • Prompt Engineering
  • AI Evaluation
  • Content Authoring
  • Technical Writing
  • Critical Thinking
  • Language Models
  • Rubric Design
  • Data Annotation
  • Quality Assurance
  • Generative AI

Tags:

Prompt Evaluator
AI evaluation
prompt engineering
content authoring
technical writing
rubric design
critical thinking
language models
data annotation
quality assurance
generative AI
AI
Large Language Models
ChatGPT
machine learning
natural language processing
data benchmarking
evaluation systems
prompt design tools
AI research platforms
content creation tools

Share Job:

How to Get Hired at North Star Staffing

  • Research North Star Staffing's focus: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor.
  • Tailor your resume: Highlight writing, critical thinking, and AI tool experience for prompt evaluation.
  • Showcase analytical skills: Prepare to demonstrate your ability to design structured instructions and rubrics.
  • Practice written assessments: Sharpen your rubric design skills for the mandatory 45-minute writing test.
  • Familiarize with AI tools: Be ready to discuss your experience with ChatGPT or similar language models.

Frequently Asked Questions

Find answers to common questions about this job opportunity

Explore similar opportunities that match your background