Prompt Evaluator

Role: Prompt Evaluator

North Star Staffing's client is seeking a Generalist Evaluator Expert to contribute to a high-impact AI research initiative. This remote, hourly contract role involves crafting structured prompt-answer pairs and developing evaluation rubrics for training and benchmarking advanced language models.

About the Role

This short-term, flexible engagement is perfectly suited for strong writers and analytical thinkers who excel at transforming intricate concepts into clear, structured instructions. You will be instrumental in designing effective prompts, defining precise grading standards, and accurately assessing AI-generated outputs within a dynamic research-driven environment.

Key Responsibilities

• Design detailed prompts incorporating multiple constraints and structured instructions.
• Author high-quality “golden answers” that align with predefined expectations.
• Develop comprehensive evaluation rubrics and clear grading criteria.
• Run prompts through AI systems and meticulously assess outputs against established standards.
• Actively participate in quality assurance cycles to ensure rigorous consistency and accuracy.
• Contribute to crucial benchmarking workflows before dataset integration.

Ideal Qualifications

• Possession of a BS or BA from a reputable institution (completed or currently in progress).
• Demonstrated strong writing, editing, and critical thinking capabilities.
• Proven ability to work independently and consistently meet structured deadlines.
• Familiarity with ChatGPT or similar AI tools, whether for personal or professional use.
• Prior experience in teaching or research is preferred.
• Applicants must be based in either the United States or Canada.

Project Details

• A commitment of at least 20 hours per week is required.
• The expected duration of the project is approximately 2 months.
• Work within a structured project environment with clearly defined goals and tooling.

Application Process

• Complete a brief AI-led interview (approximately 15 minutes).
• Undertake a 45-minute written assessment focusing on rubric design.
• Selected candidates will receive an invitation to commence project work.

More About the Opportunity

• Enjoy a fully remote position with a flexible schedule.
• Engage as an independent contractor.
• Receive weekly payments via Stripe or Wise.
• The project scope may be extended based on individual performance.

Key skills/competency

• Prompt Engineering
• AI Evaluation
• Natural Language Processing (NLP)
• Technical Writing
• Critical Thinking
• Data Annotation
• Rubric Design
• Quality Assurance
• Language Models (LLMs)
• Independent Work