AI Test Automation Engineer

Introduction

At IBM Infrastructure & Technology, we design and operate the systems that keep the world running. From high-resiliency mainframes and hybrid cloud platforms to networking, automation, and site reliability. Our teams ensure the performance, security, and scalability that clients and industries depend on every day. Working in Infrastructure & Technology means tackling complex challenges with curiosity and collaboration. You'll work with diverse technologies and colleagues worldwide to deliver resilient, future-ready solutions that power innovation. With continuous learning, career growth, and a supportive culture, IBM provides the opportunities to build expertise and shape the infrastructure that drives progress.

Your Role And Responsibilities

We are looking for a Senior QA Engineer with 5+ years of overall QA experience, a strong foundation in traditional QA automation, and hands-on exposure to Machine Learning and Generative AI systems.

This role combines classical test automation (UI, API, regression, CI/CD) with modern AI quality engineering, focusing on ensuring accuracy, reliability, safety, performance, and trust of AI-driven features in production—especially in enterprise-scale platforms.

Key Responsibilities

Traditional QA Automation & Test Engineering

• Design, develop, and maintain robust automated test suites for web, API, and backend systems
• Automate UI tests using Selenium (mandatory) with maintainable page-object or screen-play patterns
• Build and maintain API automation for REST services (integration, and regression tests using REST Assured)
• Own regression, smoke, and sanity suites with clear quality gates
• Integrate automated tests into CI/CD pipelines with reliable pass/fail signals
• Drive best practices in test pyramid, test data management, and test stability
• Analyze flaky tests and improve automation reliability and execution time

AI / ML Quality Engineering

• Test LLM and GenAI features, including prompt behavior, response quality, hallucinations, edge cases, and failure modes
• Validate RAG pipelines for retrieval accuracy, relevance, grounding, and citation correctness
• Perform ML regression testing across model versions, prompt changes, and data updates
• Design test strategies for non-deterministic outputs, including scoring thresholds, tolerance bands, and golden datasets
• Validate agentic or multi-step AI workflows where applicable

Non-Functional & Responsible AI Testing

• Validate performance characteristics: latency, throughput, concurrency, and scalability
• Test reliability and resilience under load and failure scenarios
• Validate security, privacy, and Responsible AI risks, including:

• PII leakage
• Prompt injection and jailbreak attempts
• Bias, toxicity, and unsafe outputs

• Collaborate closely with backend, ML, SRE, platform, and product teams

Required Technical And Professional Expertise

Skills & Experience

• 3-5 years of experience in software QA / test engineering
• Strong hands-on experience in traditional QA automation
• Proven experience with UI automation using Selenium (mandatory)
• Solid experience in API automation (REST Assured / JSON)
• Strong understanding of test pyramid, regression strategies, and CI/CD integration
• Core Java, Python for test automation (mandatory)
• Hands-on exposure to ML or GenAI systems testing
• Solid understanding of:

• LLMs and prompt engineering
• RAG architectures
• AI regression and model drift concepts

Preferred Skills (Added Enterprise Storage Focus)

• Experience testing enterprise storage platforms (block, file, object storage)
• Familiarity with storage concepts such as:

• IOPS, latency, throughput, queue depth
• Replication, snapshots, backups, disaster recovery
• High availability (HA), failover, and resiliency testing

• Experience testing storage management, observability, or infrastructure platforms
• Exposure to distributed systems and data consistency concepts
• Familiarity with on-prem, hybrid, or cloud storage environments
• Experience validating scale, endurance, and long-running workloads
• Experience with agentic AI frameworks (LangChain, LangGraph, etc.)
• Familiarity with vector databases, embeddings, and semantic search
• Exposure to MLOps / LLMOps practices
• Experience with BDD frameworks (Cucumber / Behave)
• Familiarity with performance testing tools (JMeter, Locust, k6)
• Experience testing enterprise-grade or regulated systems

What Sets This Role Apart

• Bridge traditional automation excellence with AI-driven quality engineering
• Apply AI testing rigor to enterprise storage and infrastructure platforms
• Define automation standards for non-deterministic AI systems
• Work on production-grade GenAI platforms, not prototypes
• Influence Responsible AI quality practices at scale

Preferred Technical And Professional Experience

• Data Analytics Knowledge: Exposure to data analytics concepts and techniques, with ability to apply data-driven insights to inform testing strategies and optimize test automation.
• Advanced Scripting Skills: Experience with advanced scripting languages, including the ability to develop and maintain complex scripts for test automation and data analysis.
• Industry Leading Practices: Familiarity with industry-leading testing practices and methodologies, including experience with testing frameworks and tools.

Key skills/competency

• AI Testing
• Test Automation
• Selenium
• REST Assured
• CI/CD Integration
• Java Programming
• Python Programming
• Generative AI (GenAI)
• Large Language Models (LLMs)
• Responsible AI