Prompt Evaluator

Prompt Evaluator

North Star Staffing is seeking a Prompt Evaluator to support a high-impact AI research initiative. This role focuses on authoring structured prompt-answer pairs and evaluation rubrics used to train and benchmark advanced language models.

This short-term, flexible engagement is ideal for strong writers and analytical thinkers who excel at transforming complex ideas into clear, structured instructions. You will be responsible for designing prompts, defining grading standards, and assessing AI-generated outputs within a dynamic research-driven project environment.

Key Responsibilities

• Design detailed prompts with multiple constraints and structured instructions.
• Author high-quality “golden answers” aligned to defined expectations.
• Develop comprehensive evaluation rubrics and grading criteria.
• Run prompts through AI systems and assess outputs against standards.
• Participate in quality assurance cycles to ensure rigor and consistency.
• Contribute to benchmarking workflows before dataset integration.

Ideal Qualifications

• BS or BA from a reputable institution (completed or in progress).
• Strong writing, editing, and critical thinking skills.
• Ability to work independently and meet structured deadlines.
• Familiarity with ChatGPT or similar AI tools for personal or professional use.
• Experience in teaching or research preferred.
• Must be based in the United States or Canada.

Project Details

• Commitment of at least 20 hours per week.
• Expected duration of approximately 2 months.
• Structured project environment with clear goals and tooling.

Application Process

• Complete a short AI-led interview (~15 minutes).
• Complete a 45-minute written assessment focused on rubric design.
• Selected candidates will be invited to begin project work.

More About the Opportunity

• Fully remote, flexible schedule.
• Independent contractor engagement.
• Weekly payments via Stripe or Wise.
• Project scope may be extended based on performance.

Key skills/competency

• AI Research
• Language Model Evaluation
• Prompt Design
• Rubric Development
• Critical Thinking
• Technical Writing
• Quality Assurance
• Data Benchmarking
• Instructional Design
• Content Analysis