11 days ago
Prompt Evaluator
Keystone Recruitment
Hybrid
Contractor
$36,000
Hybrid
Job Overview
Job TitlePrompt Evaluator
Job TypeContractor
CategoryCommerce
Experience5 Years
DegreeMaster
Offered Salary$36,000
LocationHybrid
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
Prompt Evaluator (Remote)
About the Role
This Prompt Evaluator role at Keystone Recruitment involves supporting a high-impact AI research initiative. You will focus on authoring structured prompt–answer pairs and evaluation rubrics to train and benchmark advanced language models. This flexible engagement is ideal for strong writers and analytical thinkers who can transform complex ideas into clear, structured instructions, designing prompts, defining grading standards, and assessing AI-generated outputs within a research-driven project environment.
Key Responsibilities
- Design detailed prompts with multiple constraints and structured instructions.
- Author high-quality “golden answers” aligned to defined expectations.
- Develop comprehensive evaluation rubrics and grading criteria.
- Run prompts through AI systems and assess outputs against standards.
- Participate in quality assurance cycles to ensure rigor and consistency.
- Contribute to benchmarking workflows before dataset integration.
Ideal Qualifications
- BS or BA from a reputable institution (completed or in progress).
- Strong writing, editing, and critical thinking skills.
- Ability to work independently and meet structured deadlines.
- Familiarity with ChatGPT or similar AI tools for personal or professional use.
- Experience in teaching or research preferred.
- Must be based in the United States or Canada.
Project Details
- Commitment of at least 20 hours per week.
- Expected duration of approximately 2 months.
- Structured project environment with clear goals and tooling.
Application Process
- Complete a short AI-led interview (approximately 15 minutes).
- Complete a 45-minute written assessment focused on rubric design.
- Selected candidates will be invited to begin project work.
More About the Opportunity
- Fully remote, flexible schedule.
- Independent contractor engagement.
- Weekly payments via Stripe or Wise.
- Project scope may be extended based on performance.
Key skills/competency
- Prompt Engineering
- AI Evaluation
- Content Writing
- Analytical Thinking
- Rubric Design
- Language Models
- AI Benchmarking
- Quality Assurance
- Instructional Design
- Critical Thinking
How to Get Hired at Keystone Recruitment
- Research Keystone Recruitment's culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor.
- Tailor your resume: Highlight writing, critical thinking, and AI tool experience for Prompt Evaluator roles.
- Master the written assessment: Practice rubric design and clear instruction writing to demonstrate expertise.
- Showcase AI familiarity: Discuss your experience with ChatGPT or similar models effectively in interviews.
- Emphasize remote work readiness: Detail your ability to work independently and meet deadlines.
Frequently Asked Questions
Find answers to common questions about this job opportunity
01What does a Prompt Evaluator at Keystone Recruitment do?
02What is the expected commitment for the Prompt Evaluator role?
03What qualifications are essential for the Prompt Evaluator position?
04How is the application process structured for the Prompt Evaluator role at Keystone Recruitment?
05Can international candidates apply for the Prompt Evaluator role?
06What kind of AI research initiative will the Prompt Evaluator support?
07How will I be paid as a Prompt Evaluator for Keystone Recruitment?
08What makes this Prompt Evaluator role unique for writers and analytical thinkers?
Explore similar opportunities that match your background