Healthcare Expert

Healthcare Expert (Remote)

Keystone Recruitment's client, a leading AI research organization, is actively developing advanced conversational AI systems for healthcare and medical applications. This vital role focuses on meticulously evaluating and enhancing these systems' responses to clinical and health-related inquiries. The goal is to ensure all AI outputs are accurate, exceptionally clear, safe, and fully compliant with real-world healthcare standards.

Key Responsibilities

• Write and refine prompts that effectively guide AI behavior in complex healthcare and medical scenarios.
• Evaluate AI-generated responses to healthcare-related queries for accuracy, clarity, reasoning, and completeness.
• Verify all medical claims rigorously using trusted public sources and authoritative references.
• Annotate and score model outputs by precisely identifying strengths, weaknesses, and any factual or clinical inaccuracies.
• Assess the tone, appropriateness, and completeness of responses for authentic real-world healthcare communication.
• Ensure all outputs strictly align with defined system guidelines and conversational standards.
• Apply standardized evaluation frameworks, taxonomies, and benchmarks consistently across all tasks.

Required Qualifications

• At least 5 years of professional experience specifically within the healthcare sector.
• An advanced healthcare-related degree or credential (e.g., MD, DO, RN, NP, PA, PharmD, MPH, or equivalent).
• Expertise in one or more of the following critical areas: General clinical care, Specialty medicine or surgery, Diagnostics, imaging, or laboratory medicine, Public health, healthcare systems, or healthcare administration.
• Strong familiarity with large language models (LLMs) and how they are effectively used in practice.
• Excellent written communication skills, particularly for conveying complex medical topics with precision.
• Exceptional attention to detail and a keen ability to detect subtle clinical or factual errors.

Preferred Qualifications

• Prior experience with RLHF (Reinforcement Learning from Human Feedback), AI evaluation, or data annotation.
• A background in medical writing, editing, or clinical documentation.
• Proven experience explaining intricate medical information to non-clinical audiences clearly and effectively.
• Familiarity with structured evaluation rubrics or quality scoring systems.

What Success Looks Like

• You consistently identify medical inaccuracies, unsafe reasoning, or unclear explanations in AI responses.
• Your precise feedback directly improves the reliability, clarity, and safety of healthcare-related AI outputs.
• You deliver high-quality, reproducible evaluation data that significantly strengthens AI model performance.

Contract & Payment Terms

• This is an independent contractor engagement, paid hourly.
• Fully remote with flexible scheduling to suit your availability.
• Weekly payments are processed conveniently via Stripe or Wise.
• Project scope and duration are dynamic and may vary based on performance and evolving client needs.
• No access to confidential or proprietary employer data is required for this role.
• H1-B and STEM OPT sponsorship is not available for this position.

Application Process

• Submit your comprehensive resume for initial review.
• Selected candidates will be invited to complete a short, healthcare-focused evaluation or screening form.

Key skills/competency

• Clinical Expertise
• AI Evaluation
• Medical Writing
• Prompt Engineering
• Data Annotation
• Healthcare Standards
• Large Language Models (LLMs)
• Factual Verification
• Communication Skills
• Attention to Detail