AI Bioinformatics Benchmarking Engineer
Sequencing · United States
- Hybrid
- Full-time
- $120,000 / year
- United States
Job highlights
- Build and operate AI evaluation infrastructure.
- Design and implement benchmarking systems for AI.
- Test AI outputs against genomic datasets.
- Ensure AI systems are correct and reliable.
- Partner with AI Bioinformatics Engineering Lead.
About the role
AI Bioinformatics Benchmarking Engineer
At Sequencing, you'll build and operate the evaluation and validation infrastructure that keeps our AI systems correct, reliable, and regression-protected. This is an execution-focused role: you'll design, implement, and maintain the benchmarking systems that continuously test AI outputs against curated genomic datasets, partnering closely with the AI Bioinformatics Engineering Lead to turn interpretation standards into measurable, operational systems. You are detail-oriented, have a systems-thinking mindset and you notice the one VCF out of ten thousand that doesn't look right.
The Impact
- Build and run automated evaluation pipelines for AI outputs end-to-end.
- Expand regression datasets that cover variant normalization, transcript ambiguity, and disease-level mapping.
- Execute large-scale validation runs across curated VCF datasets, track performance over time, and surface accuracy regressions the moment they appear.
- Grow a structured question bank using DV3 and related datasets, encoding the edge cases that matter most.
- Maintain ground truth datasets for pathogenic and benign variants so our benchmarks reflect real-world genomic complexity.
- Ship automated regression tests for every AI release and build dashboards that make variant-level and case-level accuracy visible to the whole team.
- Partner with engineering to wire validation directly into CI/CD, and diagnose and document failure modes as they emerge.
Dominant and Expressed Traits
- Degree in Bioinformatics, Computational Biology, Genetics, or a related field.
- Deep, hands-on experience with VCFs and real-world genomic datasets.
- Fluency with ClinVar, dbSNP, HGVS standards, and transcript mapping.
- A track record building evaluation frameworks or testing pipelines, Langfuse or similar.
- Experience evaluating LLM-based systems and standing up automated QA at scale.
- Comfort living inside modern data engineering workflows.
Key skills/competency
- Bioinformatics
- Benchmarking
- AI Evaluation
- Genomic Datasets
- VCF
- ClinVar
- dbSNP
- HGVS
- LLM QA
- Data Engineering
Skills & topics
- AI
- Bioinformatics
- Benchmarking
- Engineer
- Genomic Data
- VCF
- QA
- LLM
- Validation
- Sequencing
How to get hired
- Tailor your resume: Highlight your bioinformatics, AI evaluation, and genomic data experience.
- Showcase your skills: Emphasize experience with VCFs, ClinVar, dbSNP, and HGVS standards.
- Demonstrate system thinking: Provide examples of building evaluation frameworks or QA at scale.
- Quantify your impact: Use metrics to show how your work improved AI accuracy or reliability.
- Prepare for technical questions: Be ready to discuss your experience with data engineering workflows and LLM evaluation.
Technical preparation
Master VCF file formats and genomic data.,Practice with ClinVar, dbSNP, and HGVS.,Build mock evaluation pipelines.,Review CI/CD integration concepts.
Behavioral questions
Describe a time you found a subtle error.,How do you approach complex system testing?,Tell me about building automated QA.,How do you collaborate with engineering teams?
Frequently asked questions
- What is the primary focus of the AI Bioinformatics Benchmarking Engineer role at Sequencing?
- The primary focus of the AI Bioinformatics Benchmarking Engineer role at Sequencing is to build and operate the evaluation and validation infrastructure for AI systems, ensuring their correctness, reliability, and regression protection using curated genomic datasets.
- What specific technical skills are most important for an AI Bioinformatics Benchmarking Engineer at Sequencing?
- Key technical skills include deep hands-on experience with VCFs and genomic datasets, fluency with ClinVar, dbSNP, HGVS standards, and transcript mapping, experience building evaluation frameworks or testing pipelines (like Langfuse), and comfort with modern data engineering workflows and LLM evaluation.
- What kind of datasets will I be working with as an AI Bioinformatics Benchmarking Engineer at Sequencing?
- You will work with curated genomic datasets, including VCFs, ground truth datasets for pathogenic and benign variants, and datasets for variant normalization, transcript ambiguity, and disease-level mapping. You will also expand regression datasets and grow a structured question bank using DV3 and related datasets.
- How does this role contribute to the overall AI development at Sequencing?
- This role is crucial for ensuring the quality and reliability of Sequencing's AI systems. By building and operating robust benchmarking and validation infrastructure, you'll directly contribute to shipping automated regression tests for AI releases and making accuracy visible to the entire team.
- What does Sequencing look for in terms of personal attributes for this role?
- Sequencing looks for individuals who are detail-oriented, possess a systems-thinking mindset, and are proactive in identifying issues, such as spotting an incorrect VCF among thousands. A strong degree in a relevant field like Bioinformatics or Computational Biology is also expected.
- Can you provide examples of how an AI Bioinformatics Benchmarking Engineer might expand regression datasets?
- Expanding regression datasets might involve creating new test cases that specifically cover challenging scenarios like complex variant normalization issues, ambiguous transcript mappings, or inaccuracies in disease-level variant interpretation. The goal is to ensure comprehensive test coverage for AI outputs.
- What is the expected level of collaboration in this AI Bioinformatics Benchmarking Engineer role?