Healthcare Expert

Role Overview

The Healthcare Expert role at Keystone Recruitment focuses on evaluating and improving advanced conversational AI systems used in healthcare and medical contexts. This independent contractor position requires ensuring outputs are accurate, clear, safe, and aligned with real-world healthcare standards.

Key Responsibilities

• Write and refine prompts guiding AI behavior in healthcare scenarios.
• Evaluate AI-generated responses for accuracy, clarity, reasoning, and completeness.
• Verify medical claims using trusted sources and authoritative references.
• Annotate and score model outputs by identifying strengths, weaknesses, and inaccuracies.
• Assess tone, appropriateness, and completeness for communication.
• Ensure all outputs align with defined system guidelines and conversational standards.
• Apply standardized evaluation frameworks, taxonomies, and benchmarks consistently.

Required Qualifications

At least 5 years of professional healthcare experience with an advanced healthcare-related degree or credential (e.g., MD, DO, RN, NP, PA, PharmD, MPH, or equivalent). Must have expertise in general clinical care, specialty medicine, diagnostics, public health, or healthcare administration, and strong familiarity with large language models (LLMs) in practice. Excellent written communication and attention to detail are essential.

Preferred Qualifications

Experience with RLHF, AI evaluation or data annotation, medical writing or clinical documentation, and familiarity with structured evaluation rubrics.

What Success Looks Like

You will consistently identify medical inaccuracies, unsafe reasoning, or unclear explanations. Your feedback will improve the reliability, clarity, and safety of healthcare AI outputs, delivering high-quality evaluation data that strengthens the AI model performance.

Contract & Payment Terms

This is an independent contractor, fully remote position with flexible scheduling. Payment is hourly via Stripe or Wise with weekly payments. The project scope and duration may vary based on performance and client needs. Note that H1-B and STEM OPT sponsorship is not available.

Application Process

Submit your resume for review. Selected candidates will complete a short healthcare-focused evaluation or screening form.

Key skills/competency

• Healthcare
• AI
• Evaluation
• Annotation
• Clinical
• Medical
• Communication
• Attention
• Accuracy
• Guidelines