Senior QA Engineer, AI/ML

Senior QA Engineer, AI/ML at Autodesk

The Applied AI team in Autodesk's Data and Process Management (DPM) organization ships cloud-native services that power AI agents and AI-driven workflows that make our Product Data Management (PDM) and Product Lifecycle Management (PLM) workflows smarter and easier.

We’re hiring a Senior QA Engineer, AI/ML who is automation-first and production-minded. You’ll start by owning quality for cloud services and AI-integrated features, then grow into owning our AI evaluation pipelines (built on internal frameworks and Opik) over the next 2–3 quarters.

You don’t need to be an ML expert on day one—but you do need strong software engineering fundamentals, comfort working with distributed systems, and curiosity to learn AI-specific quality evaluation patterns and tools.

Responsibilities

• Build and maintain automated tests for cloud-native services: API/contract tests, and end-to-end workflow tests
• Validate non-functional requirements: performance, resiliency/failure modes, multi-tenant behavior, and observability-driven debugging (logs/metrics/traces)
• Partner with engineers and PMs to define acceptance criteria and quality gates for releases
• Develop and maintain scenario-based regression suites for AI-integrated workflows (multi-step tasks, tool calls, retrieval-backed behaviors)
• Build and operationalize evaluation pipelines using internal frameworks and evaluation tools like Opik
• Curate and maintain “golden” datasets (test cases, expected behaviors, labels/metadata)
• Automate Agent evaluation runs (CI, scheduled runs, and/or sampled runtime evaluation)
• Publish results to dashboards and establish alerting for failures/regressions
• Security-aware testing for AI surfaces: include abuse cases (e.g., prompt-injection style attempts, unsafe tool execution paths, sensitive-data leakage checks) and verify guardrails/controls
• AI-assisted delivery: Use AI coding agents to accelerate delivery of tests and automations

Minimum Qualifications

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or equivalent practical experience
• 4+ years experience as a QA Engineer, SDET, or Software Engineer with substantial test automation ownership
• Strong programming skills in Python and/or TypeScript/Java; you write maintainable automation code, not just scripts
• Experience testing cloud-native distributed systems (REST/GraphQL APIs, async workflows, service-to-service integrations)
• Proven verification habits: test design, CI hygiene, disciplined incremental delivery, and strong debugging skills
• Comfort operating production-like systems: reading telemetry, reproducing issues, triaging failures, and driving fixes with engineers
• Strong communication: you can document test strategy, influence quality gates, and collaborate cross-functionally
• Experience with testing AI-integrated systems in production (any of): LLM feature regression testing, prompt/version change validation RAG-style workflows (retrieval quality checks, grounding/citation checks, data freshness)
• Tool-use / agentic workflows (validating tool-call sequences and failure recovery paths)
• Demonstrated experience using AI coding tools to develop tests for production systems, and the engineering judgment to verify and correct AI output (code review rigor, debugging skill, ownership of correctness)

Preferred Qualifications

• Familiarity with evaluation tooling (Opik, Langfuse, or similar), dataset versioning practices, and automated evaluation runs
• Experience with performance testing and resiliency patterns (rate limiting, retries/idempotency validation, chaos/fault testing)
• Security-minded testing experience, especially for systems that integrate external tools/data sources

Key skills/competency

• Test Automation
• Cloud-Native Services
• AI Evaluation
• Distributed Systems
• Python/TypeScript/Java
• API/Contract Testing
• Performance Testing
• Security Testing
• Debugging
• CI/CD