AI Evaluation Lead

About the Role

At BlueCross BlueShield of Tennessee, we are building a member-facing virtual AI assistant to improve access to information and streamline the healthcare journey. As the AI Evaluation Lead, you will own the evaluation strategy and dataset management to ensure our assistant remains safe, trustworthy, and high-quality.

Key Responsibilities

• Develop and operationalize evaluation and quality programs for AI features.
• Define success criteria, acceptance thresholds, and release readiness gates.
• Establish scalable evaluation frameworks combining functional, non-functional, and user experience tests.
• Translate stakeholder needs into measurable requirements and validation suites.
• Partner with engineering to integrate automated testing into CI/CD pipelines.
• Track quality metrics and drive continuous improvement.
• Facilitate defect triage and communicate quality risks.
• Contribute to compliance, risk assessments, and documentation practices.
• Mentor team members on evaluation design and operational excellence.

Job Qualifications

• Bachelor's in STEM or equivalent work experience (Master’s/PhD preferred).
• 5+ years in technology, software delivery, analytics, or QA.
• 3+ years with SDLC and formal quality management practices.
• Experience in automated testing, defect tracking, and cloud environments.
• Proficiency in Python and analytics tooling.
• Experience evaluating AI/ML-enabled systems is a plus.

Additional Information

This is a fully remote role; onsite interviews at our Chattanooga, TN headquarters may be required. Sponsorship is not available for this role.

Key skills/competency

• Evaluation
• Quality Assurance
• AI
• Python
• SDLC
• Automation
• Data Curation
• Testing
• Compliance
• Analytics