Prompt Evaluator

Prompt Evaluator

Keystone Recruitment is seeking a Prompt Evaluator to support a high-impact AI research initiative for one of their clients. This hourly contract role is fully remote and focuses on authoring structured prompt–answer pairs and evaluation rubrics, essential for training and benchmarking advanced language models.

This short-term, flexible engagement is ideal for strong writers and analytical thinkers who excel at transforming complex ideas into clear, structured instructions. You will be responsible for designing prompts, defining rigorous grading standards, and assessing AI-generated outputs within a dynamic research-driven project environment.

Key Responsibilities

• Design detailed prompts with multiple constraints and structured instructions.
• Author high-quality “golden answers” aligned with defined expectations.
• Develop comprehensive evaluation rubrics and grading criteria.
• Run prompts through AI systems and meticulously assess outputs against established standards.
• Participate in quality assurance cycles to ensure rigor and consistency.
• Contribute to benchmarking workflows before dataset integration.

Ideal Qualifications

• BS or BA from a reputable institution (completed or in progress).
• Strong writing, editing, and critical thinking skills.
• Ability to work independently and meet structured deadlines.
• Familiarity with ChatGPT or similar AI tools for personal or professional use.
• Experience in teaching or research preferred.
• Must be based in the United States or Canada.

Project Details

• Commitment of at least 20 hours per week.
• Expected duration of approximately 2 months.
• Structured project environment with clear goals and tooling.

Application Process

• Complete a short AI-led interview (approximately 15 minutes).
• Complete a 45-minute written assessment focused on rubric design.
• Selected candidates will be invited to begin project work.

More About the Opportunity

• Fully remote, flexible schedule.
• Independent contractor engagement.
• Weekly payments via Stripe or Wise.
• Project scope may be extended based on performance.

Key skills/competency

• Prompt Engineering
• AI Evaluation
• Technical Writing
• Critical Thinking
• Language Models
• Rubric Design
• Data Annotation
• Quality Assurance
• AI Research
• Analytical Skills