Prompt Evaluator

About the Prompt Evaluator Role at North Star Staffing

North Star Staffing is seeking a Generalist Evaluator Expert to join a high-impact AI research initiative. This remote, hourly contract role focuses on authoring structured prompt-answer pairs and developing robust evaluation rubrics. These critical contributions will be used to train and benchmark advanced language models, playing a direct role in shaping the future of AI technology.

What You Will Do

This short-term, flexible engagement is perfectly suited for individuals with strong writing and analytical skills who can transform complex ideas into clear, structured instructions. You will be instrumental in designing effective prompts, defining precise grading standards, and rigorously assessing AI-generated outputs within a dynamic, research-driven project environment.

Key Responsibilities

• Design detailed prompts with multiple constraints and structured instructions.
• Author high-quality “golden answers” that align with defined expectations.
• Develop comprehensive evaluation rubrics and precise grading criteria.
• Run prompts through AI systems and meticulously assess outputs against established standards.
• Participate in quality assurance cycles to ensure the rigor and consistency of evaluations.
• Contribute to benchmarking workflows before dataset integration, ensuring data quality.

Ideal Qualifications

• A Bachelor of Science (BS) or Bachelor of Arts (BA) degree (completed or currently in progress) from a reputable institution.
• Demonstrated strong writing, editing, and critical thinking abilities.
• Proven capacity to work independently and consistently meet structured deadlines.
• Familiarity with ChatGPT or other similar AI tools, whether for personal or professional use.
• Prior experience in teaching or research is preferred.
• Must be legally based in the United States or Canada.

Project Details

• Commitment of at least 20 hours per week.
• Expected duration of approximately 2 months, with potential for extension.
• Work within a structured project environment with clear goals and established tooling.

How to Apply

The application process involves two main steps:

• A short AI-led interview (approximately 15 minutes).
• A 45-minute written assessment focusing on rubric design.

Selected candidates who successfully complete these steps will be invited to begin project work.

More About This Opportunity

• Fully remote position offering a flexible schedule.
• Engaged as an independent contractor.
• Receive weekly payments conveniently via Stripe or Wise.
• Project scope may be extended based on performance and project needs.

Key skills/competency

• Prompt Engineering
• Content Evaluation
• AI Model Training
• Natural Language Processing
• Critical Thinking
• Technical Writing
• Instruction Design
• Quality Assurance
• Research Methodology
• Analytical Skills