Prompt Evaluator
North Star Staffing
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
Prompt Evaluator (Remote)
North Star Staffing is seeking a Generalist Evaluator Expert to join a high-impact AI research initiative. This hourly contract role is fully remote and offers a flexible schedule, ideal for strong writers and analytical thinkers eager to shape advanced language models. You will be instrumental in authoring structured prompt-answer pairs and developing robust evaluation rubrics.
About the Prompt Evaluator Role
This short-term, flexible engagement empowers you to transform complex ideas into clear, structured instructions. You will design intricate prompts, define precise grading standards, and critically assess AI-generated outputs within a dynamic research environment. The project is structured with clear goals and tooling, and offers weekly payments via Stripe or Wise, with potential for extension based on performance.
Key Responsibilities
- Design detailed prompts with multiple constraints and structured instructions.
- Author high-quality “golden answers” aligned to defined expectations.
- Develop comprehensive evaluation rubrics and grading criteria.
- Run prompts through AI systems and assess outputs against established standards.
- Participate in quality assurance cycles to ensure rigor and consistency.
- Contribute to benchmarking workflows before dataset integration.
Ideal Qualifications
- BS or BA from a reputable institution (completed or in progress).
- Strong writing, editing, and critical thinking skills.
- Ability to work independently and meet structured deadlines.
- Familiarity with ChatGPT or similar AI tools for personal or professional use.
- Experience in teaching or research is preferred.
- Must be based in the United States or Canada.
Project Details
- Commitment of at least 20 hours per week.
- Expected duration of approximately 2 months.
- Structured project environment with clear goals and tooling.
Application Process
- Complete a short AI-led interview (~15 minutes).
- Complete a 45-minute written assessment focused on rubric design.
- Selected candidates will be invited to begin project work.
Key skills/competency
- Prompt Design
- AI Evaluation
- Rubric Development
- Critical Thinking
- Technical Writing
- Language Models
- Data Annotation
- Quality Assurance
- Research Analysis
- ChatGPT
How to Get Hired at North Star Staffing
- Research North Star Staffing's mission: Study their approach to staffing and their client-focused values, examining their LinkedIn for recent news.
- Tailor your resume for prompt evaluation: Highlight your writing, critical thinking, and AI tool experience using keywords like 'prompt engineering,' 'rubric development,' and 'AI assessment.'
- Excel in the AI-led interview: Practice concise, articulate responses demonstrating your analytical skills and understanding of AI concepts.
- Master the written assessment: Showcase your ability to design clear, structured rubrics and 'golden answers' that reflect the role's core demands.
- Emphasize independent work and deadlines: During interviews, provide examples of your self-management and reliability in meeting project timelines.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background