Senior QA Automation Engineer, AI Platforms

Position Overview

As a Senior QA Automation Engineer, AI Platforms at Deutsche Bank, you will be instrumental in ensuring the quality, reliability, and stability of AI-powered applications and platforms. This role involves designing robust automation frameworks, validating machine-learning-driven outputs, and ensuring consistent performance of both traditional and AI-driven systems. You will collaborate closely with developers, data scientists, ML engineers, and product teams to build comprehensive test strategies that validate model behavior, data quality, and AI user experiences.

Your Key Responsibilities

Core QA Responsibilities

• Design and maintain automation frameworks for UI, API, and backend testing.
• Develop test automation scripts using Selenium, Playwright, Cypress, or similar.
• Build API automation suites using RestAssured, Postman/Newman, etc.
• Perform test planning, execution, reporting, and defect lifecycle management.
• Implement automated tests in CI/CD pipelines (Jenkins, TeamCity, GitHub Actions).
• Perform root-cause analysis and support L3 production issues.

AI-Focused QA Responsibilities

• Collaborate with Data Science and ML teams to test AI/ML model behavior.
• Validate model outputs for accuracy, precision, recall, thresholds, and stability.
• Test AI-driven features: recommendations, NLP/chatbots, classification, anomaly detection, predictive insights, etc.
• Automate validation of model inference APIs, streaming outputs, and batch pipelines.
• Conduct data-quality testing for ML feature inputs (schema, drift, distribution checks).
• Validate model retraining, deployment workflows, and versioning pipelines.
• Test UI interfaces showing model predictions, confidence scores, and explainability outputs.
• Ensure fairness, bias detection, and responsible AI presentation in product workflows.
• Collaborate with MLOps to test end-to-end ML lifecycle.

Tools & Frameworks for AI Model Testing

AI/ML Testing Frameworks

• Deepchecks – ML model validation, data integrity, drift checks
• DeepEval / TruEra / Arthur AI – model evaluation, quality monitoring, bias & fairness checks
• Evidently AI – automated monitoring for model drift, data drift, data quality
• Fiddler AI – model explainability, fairness, performance dashboards
• MLflow – model validation, experiment tracking, test comparison
• Great Expectations – data validation for ML pipelines

Data & Pipeline Testing

• PyTest + custom ML test harnesses
• Soda Data / Deequ – data quality checks for ML features
• Airflow/Kubeflow test modules – pipeline DAG testing

Model Inference / API Testing Tools

• Postman/Newman – testing ML inference endpoints
• RestAssured – API-based model validation
• Locust/JMeter – load testing of inference throughput & latency

Observability & Monitoring Tools

• Prometheus + Grafana – monitoring inference latency & performance
• Elastic Stack (ELK) – logging of model behavior & anomalies
• OpenTelemetry – tracing AI pipeline performance

Your Skills And Experience

Technical Skills

• Test automation using Selenium, Playwright, Cypress, or equivalent.
• Strong programming/scripting in Python, Java, or JavaScript.
• Framework design experience (POM, Hybrid, BDD, Data-Driven).
• Strong background in API testing.
• Experience with ML/AI testing frameworks (Evidently, Deepchecks, MLflow, Fiddler, GE).
• Understanding of ML workflows, model metrics, and data validation.
• Experience with cloud platforms and containers.

Soft Skills

• Strong analytical and debugging skills.
• Attention to detail and quality mindset.
• Ability to collaborate with engineering, AI, and product teams.
• Clear communication skills; ability to explain ML test findings.

How We’ll Support You

• Training and development to help you excel in your career.
• Coaching and support from experts in your team.
• A culture of continuous learning to aid progression.
• A range of flexible benefits that you can tailor to suit your needs.

Key skills/competency

• QA Automation
• AI Testing
• ML Model Validation
• Python/Java/JavaScript
• Test Framework Design
• API Testing
• CI/CD
• Data Quality
• Selenium/Playwright/Cypress
• Deepchecks/MLflow