Prompt Evaluator

Prompt Evaluator at North Star Staffing

Role Summary

North Star Staffing is seeking a Generalist Evaluator Expert for a high-impact AI research initiative. This remote, hourly contract role involves authoring structured prompt–answer pairs and evaluation rubrics to train and benchmark advanced language models.

About the Role

This short-term, flexible engagement is perfect for strong writers and analytical thinkers eager to transform complex ideas into clear, structured instructions. You will design AI prompts, define grading standards, and meticulously assess AI-generated outputs within a dynamic, research-driven project environment.

Key Responsibilities

• Design detailed prompts with multiple constraints and structured instructions.
• Author high-quality “golden answers” that align with defined expectations.
• Develop comprehensive evaluation rubrics and precise grading criteria.
• Run prompts through various AI systems and rigorously assess outputs against established standards.
• Participate in quality assurance cycles to ensure project rigor and consistency.
• Contribute actively to benchmarking workflows before final dataset integration.

Ideal Qualifications

• Bachelor of Science or Bachelor of Arts degree (completed or in progress) from a reputable institution.
• Demonstrated strong writing, editing, and critical thinking skills.
• Proven ability to work independently and consistently meet structured deadlines.
• Familiarity with ChatGPT or similar AI tools for personal or professional use.
• Experience in teaching or research is preferred.
• Must be based in the United States or Canada.

Project Details

• Commitment of at least 20 hours per week.
• Expected duration of approximately 2 months.
• Structured project environment with clear goals and tooling.

Application Process

• Complete a short AI-led interview (approximately 15 minutes).
• Complete a 45-minute written assessment focused on rubric design.
• Selected candidates will be invited to begin project work directly.

More About the Opportunity

• Fully remote position with a flexible schedule.
• Engage as an independent contractor.
• Receive weekly payments via Stripe or Wise.
• Project scope may be extended based on performance and project needs.

Key skills/competency

• AI Evaluation
• Prompt Design
• Rubric Development
• Content Writing
• Analytical Thinking
• Language Models
• Quality Assurance
• Critical Thinking
• Data Benchmarking
• Instruction Design