Prompt Evaluator

Prompt Evaluator at North Star Staffing

North Star Staffing is seeking a dedicated Prompt Evaluator to join a high-impact AI research initiative. This remote, hourly contract role focuses on crafting structured prompt–answer pairs and developing comprehensive evaluation rubrics to train and benchmark advanced language models.

This short-term, flexible engagement is ideal for individuals with strong writing and analytical skills who can transform complex concepts into clear, structured instructions. You will be responsible for designing prompts, defining grading standards, and meticulously assessing AI-generated outputs within a dynamic, research-driven project environment.

Key Responsibilities

• Design detailed prompts incorporating multiple constraints and structured instructions.
• Author high-quality “golden answers” that align with predefined expectations.
• Develop comprehensive evaluation rubrics and precise grading criteria.
• Run prompts through various AI systems and rigorously assess their outputs against established standards.
• Participate actively in quality assurance cycles to ensure project rigor and consistency.
• Contribute to benchmarking workflows before the final integration of datasets.

Ideal Qualifications

• Bachelor's or Associate's degree from a reputable institution (completed or in progress).
• Demonstrated strong writing, editing, and critical thinking skills.
• Proven ability to work independently and adhere to structured deadlines.
• Familiarity with ChatGPT or similar AI tools for either personal or professional use.
• Experience in teaching or research is preferred.
• Must be based in the United States or Canada.

Project Details

• A commitment of at least 20 hours per week.
• Expected duration of approximately 2 months.
• Work within a structured project environment with clear goals and tooling.

Application Process

• Complete a short AI-led interview (approximately 15 minutes).
• Successfully complete a 45-minute written assessment focused on rubric design.
• Selected candidates will be invited to commence project work.

More About the Opportunity

• Fully remote position with a flexible schedule.
• Independent contractor engagement.
• Weekly payments processed via Stripe or Wise.
• Project scope may be extended based on performance and project needs.

Key skills/competency

• Prompt Engineering
• AI Evaluation
• Language Models
• Technical Writing
• Critical Thinking
• Analytical Skills
• Rubric Development
• Quality Assurance
• ChatGPT
• Independent Work