AI Content Evaluator

AI Content Evaluator

A leading AI research initiative at The UVA VEC is seeking detail-oriented writing professionals for the role of AI Content Evaluator. This position is crucial for supporting the development and evaluation of advanced language models. As an AI Content Evaluator, you will be responsible for creating structured prompt-and-answer content to train AI systems, ensuring they produce accurate, helpful, and high-quality responses. This flexible, short-term contract opportunity is ideal for individuals who excel at simplifying complex ideas and establishing clear evaluation standards for real-world applications. Applicants should possess a strong academic background and excellent written communication skills, and be based in the United States or Canada.

Key Responsibilities

• Design detailed prompts with clear constraints and instructions.
• Develop evaluation frameworks and response quality standards.
• Assess AI-generated outputs for accuracy and clarity.
• Establish grading rubrics for expected model responses.
• Support benchmarking and quality assurance processes.
• Collaborate on ensuring consistency across prompt datasets.

Minimum Qualifications

• Bachelor’s degree (completed or in progress).
• Strong writing and critical thinking skills.
• Ability to work independently and meet deadlines.
• Familiarity with AI tools such as ChatGPT for everyday use.
• Based in the United States or Canada.

Preferred Qualifications

• Background in teaching, research, or instructional writing.
• Experience developing evaluation criteria or rubrics.

Project Details

• Expected commitment: approximately 20 hours per week.
• Project duration: approximately 2 months.
• Structured workflow with defined tools and goals.

Application Process

• Complete a short AI-led interview (~15 minutes).
• Complete a 45-minute written rubric assessment.
• Selected candidates will be invited to join the project.

Contract & Payment Terms

• Independent contractor engagement.
• Fully remote with flexible scheduling.
• Project duration may vary based on performance and needs.
• Weekly payments via Stripe or Wise.
• Work does not involve access to confidential third-party information.

Key skills/competency

• AI content evaluation
• Prompt engineering
• Language models
• Content accuracy
• Writing skills
• Critical thinking
• Rubric development
• Quality assurance
• Benchmarking
• AI tools familiarity