Applied Scientist, Artificial General Intelligence
Amazon · Bellevue, WA
- On site
- Full-time
- $184,000 / year
- Bellevue, WA
Job highlights
- Develop LLM quality strategies and auditing frameworks.
- Design and build LLM-as-a-Judge systems.
- Conduct manual and meta-audits for data quality.
- Collaborate with scientists and engineers on Nova models.
- Enhance customer experience via high-quality data.
About the role
Applied Scientist, Artificial General Intelligence
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.
Key job responsibilities
As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
A day in a life
An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
Basic Qualifications
- Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field
- Experience programming in Java, C++, Python or related language
- Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse
Preferred Qualifications
- Experience implementing algorithms using both toolkits and self-developed code
- Have publications at top-tier peer-reviewed conferences or journals
Key skills/competency
- Artificial General Intelligence
- Machine Learning
- LLMs
- Data Quality
- Auditing Methodologies
- Automated Evaluation
- Statistics
- Python
- SQL
- Algorithm Implementation
Skills & topics
- Applied Scientist
- Artificial General Intelligence
- AGI
- Machine Learning
- LLM
- Data Quality
- Auditing
- Automated Evaluation
- Python
- SQL
- Amazon
- Computer Science
- Statistics
How to get hired
- Tailor your resume: Highlight your Master's degree, programming (Python, Java, C++), SQL, and machine learning expertise.
- Showcase AI/ML achievements: Emphasize experience with LLMs, auditing, and automated evaluation systems.
- Prepare for technical interviews: Brush up on algorithms, statistics, and RDBMS concepts.
- Demonstrate problem-solving skills: Be ready to discuss root cause analysis and quality improvement strategies.
- Research Amazon's culture: Understand their focus on customer obsession and innovation.
Technical preparation
Behavioral questions
Frequently asked questions
- What specific experience is required for the Applied Scientist AGI role at Amazon?
- The Applied Scientist role at Amazon requires a Master's degree in a quantitative field like computer science or machine learning, proficiency in programming languages such as Python, Java, or C++, and experience with SQL and relational databases. Experience with auditing methodologies and automated evaluation systems is also crucial for this position within the Artificial General Intelligence team.
- What are the key responsibilities of an Applied Scientist on the AGI team at Amazon?
- An Applied Scientist on Amazon's AGI team is responsible for developing comprehensive quality strategies and auditing frameworks for LLMs and multimodal systems. This includes designing auditing procedures, creating LLM-as-a-Judge systems, performing manual audits, and collaborating with scientists and engineers to ensure data integrity and improve model performance.
- Does Amazon offer workplace accommodations for applicants with disabilities for the Applied Scientist role?
- Yes, Amazon is committed to providing reasonable workplace accommodations for applicants with disabilities. If you need any assistance during the application, hiring, or onboarding process for the Applied Scientist AGI position, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information or contact your Recruiting Partner.
- What is the expected salary range for the Applied Scientist AGI position at Amazon in Boston or Bellevue?
- The base salary range for the Applied Scientist Artificial General Intelligence position at Amazon in Boston, MA, and Bellevue, WA, is $136,000.00 to $184,000.00 USD annually. This range does not include potential sign-on payments, restricted stock units (RSUs), or comprehensive benefits.
- What are the preferred qualifications for an Applied Scientist in Amazon's AGI team?
- Preferred qualifications for an Applied Scientist on Amazon's AGI team include experience implementing algorithms using both toolkits and self-developed code, as well as having publications at top-tier peer-reviewed conferences or journals. These demonstrate a strong research and development capability.
- How does Amazon ensure fairness and non-discrimination in its hiring process for roles like Applied Scientist AGI?
- Amazon is an equal opportunity employer and does not discriminate based on protected veteran status, disability, or any other legally protected status. They strive to foster an inclusive culture where all employees can deliver their best results.