Applied Scientist, Artificial General Intelligence

About the Role

This is currently a 12-month temporary contract opportunity with the possibility to extend to 24 months based on business needs.

The Artificial General Intelligence (AGI) team at Amazon is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems. This role is crucial for ensuring the highest standards of data quality and building industry-leading technology with Large Language Models (LLMs) and multimodal systems.

Key Responsibilities

• Collaborate closely with the core scientist team developing Amazon Nova models.
• Lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
• Design auditing strategies with detailed Standard Operating Procedures (SOPs), quality metrics, and sampling methodologies to improve Nova model performances on benchmarks.
• Perform expert-level manual audits and conduct meta-audits to evaluate auditor performance.
• Provide targeted coaching to uplift overall quality capabilities.
• Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
• Set up the configuration of data collection workflows and communicate quality feedback to stakeholders.
• Directly impact enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.

A Day in the Life

An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, you will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.

Basic Qualifications

• Master's degree in engineering, statistics, computer science, mathematics, or a related quantitative field.
• 2+ years of machine learning, statistical modeling, data mining, and analytics techniques experience.
• 3+ years of programming in Java, C++, Python, or related language experience.
• 2+ years of building machine learning models or developing algorithms for business application experience.

Preferred Qualifications

• Ph.D. in engineering, technology, computer science, machine learning, robotics, operations research, statistics, mathematics, or equivalent quantitative field.
• Publications at top-tier peer-reviewed conferences or journals.
• 4+ years of solving business problems through machine learning, data mining, and statistical algorithms experience.
• Experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution, or experience debugging, profiling, and implementing best software engineering practices in large-scale systems.

Key skills/competency

• Machine Learning
• Large Language Models (LLMs)
• Multimodal Systems
• Statistical Modeling
• Quality Assurance
• Auditing Methodologies
• Automated Evaluation
• Data Quality
• Python Programming
• Algorithm Development