QA Engineer - AI Experience Essential
Project Foundry Resourcing Services · EMEA
- Hybrid
- Contract
- $90,000 / year
- EMEA
Job highlights
- Test AI/LLM products for quality and accuracy.
- Validate AI responses against data sources.
- Develop and execute manual and automated tests.
- Collaborate with cross-functional teams on AI features.
- Ensure AI interactions are reliable and secure.
About the role
QA Engineer AI
About the Role
We are seeking a QA / Test Engineer to join our dynamic and fast-paced team. This role is critical to ensuring the quality, accuracy, and reliability of our multi-agent LLM product, which powers AI-driven interactions across our business. As part of this high-impact team, the role will focus on front-end testing, answer validation, and ensuring that responses align with the correct data sources. They will work closely with developers, data scientists, and product managers to create and execute test plans that validate both functionality and response accuracy. This is a fast-paced, complex, and evolving environment where agility, attention to detail, and strong problem-solving skills are essential.
Key Responsibilities
Test Planning & Execution
- Develop and execute test plans and test cases for the AI Assistant's front-end UI and conversational interfaces.
- Create and maintain Jira tickets to development team. Strong communication skills required.
- Validate LLM-generated responses against structured and unstructured data sources to ensure accuracy and reliability.
- Perform manual and automated testing to evaluate system performance and usability.
- Collaborate with developers to troubleshoot and reproduce reported issues.
AI Response Validation & Accuracy Testing
- Implement rigorous answer validation frameworks to ensure AI-generated responses align with source data and organizational knowledge.
- Identify, document, and escalate inconsistencies, hallucinations, and gaps in AI reasoning.
Automation & Efficiency
- Develop test scripts for UI testing, and UI performance.
- Contribute to the continuous improvement of automated testing frameworks.
Performance & Security Testing
- Assess latency, scalability, and response speed of LLM interactions under different workloads.
- Work with security teams to validate data privacy, compliance, and access control in AI interactions.
Collaboration & Reporting
- Provide clear, structured defect reports and track issues from identification to resolution.
- Work closely with data scientists, AI engineers, and DevOps teams to continuously refine model behavior and system robustness.
- Participate in agile ceremonies (standups, sprint planning, retrospectives) and contribute to release readiness discussions.
Required Skills & Experience
- Proven experience in software testing and QA methodologies (manual and automated).
- Hands-on experience testing AI/LLM products, chatbots, or NLP-driven applications is a must.
- Strong understanding of UI, functional, regression, and exploratory testing techniques.
- Basic understanding of LLMs, NLP models, and AI response validation is required.
- Ability to identify AI hallucinations, errors, and inconsistencies in generated responses.
- Strong attention to detail and a methodical approach to troubleshooting issues.
- Experience in a fast-paced, high-pressure environment with shifting priorities.
- Ability to work across multiple projects and prioritize based on impact and urgency.
- Strong written and verbal communication skills to document test results and defects.
- Ability to work effectively across engineering, product, and delivery teams.
Key skills/competency
- AI Testing
- LLM Products
- NLP Applications
- Front-end Testing
- Answer Validation
- Automated Testing
- Performance Testing
- Security Testing
- Agile Methodologies
- Jira
Skills & topics
- QA Engineer
- AI Testing
- LLM
- NLP
- Software Testing
- Automated Testing
- Front-end Testing
- Performance Testing
- Security Testing
- Jira
How to get hired
- Tailor your resume: Highlight AI/LLM testing experience and QA methodologies.
- Showcase AI expertise: Detail your understanding of LLMs and NLP in your application.
- Prepare for technical questions: Be ready to discuss AI response validation and automation.
- Demonstrate soft skills: Emphasize collaboration, communication, and problem-solving abilities.
- Network effectively: Connect with Project Foundry Resourcing Services team members on LinkedIn.
Technical preparation
Practice AI response validation techniques.,Build UI test scripts for LLM interfaces.,Familiarize with NLP model concepts.,Set up automated testing frameworks.
Behavioral questions
Describe a complex AI bug you found.,How do you handle shifting project priorities?,Explain your approach to collaborative problem-solving.,How do you ensure attention to detail?
Frequently asked questions
- What is the primary focus for a QA Engineer AI at Project Foundry Resourcing Services?
- The primary focus for a QA Engineer AI at Project Foundry Resourcing Services is to ensure the quality, accuracy, and reliability of their multi-agent LLM product, specifically testing its front-end UI, conversational interfaces, and validating AI-generated responses against data sources.
- Is hands-on experience with AI/LLM products a strict requirement for this QA Engineer role?
- Yes, hands-on experience testing AI/LLM products, chatbots, or NLP-driven applications is explicitly stated as a must-have requirement for this QA Engineer AI position.
- What kind of testing is involved in the QA Engineer AI role at Project Foundry Resourcing Services?
- The QA Engineer AI role involves a mix of manual and automated testing, including UI testing, functional testing, regression testing, exploratory testing, performance testing, and security testing.
- How does Project Foundry Resourcing Services ensure the accuracy of AI-generated responses?
- Project Foundry Resourcing Services implements rigorous answer validation frameworks, comparing AI-generated responses against structured and unstructured data sources to ensure accuracy and identify inconsistencies or hallucinations.
- What technical skills are essential for the QA Engineer AI position?
- Essential technical skills include proven software testing methodologies, hands-on AI/LLM product testing, understanding of LLMs/NLP, ability to identify AI hallucinations, and experience with automated testing frameworks.
- What is the work environment like at Project Foundry Resourcing Services for this role?
- The work environment is described as dynamic, fast-paced, complex, and evolving, requiring agility, attention to detail, and strong problem-solving skills, often involving shifting priorities across multiple projects.
- Does the QA Engineer AI role involve collaboration with other teams at Project Foundry Resourcing Services?
- Yes, the role involves close collaboration with developers, data scientists, product managers, AI engineers, and DevOps teams to refine model behavior, ensure system robustness, and participate in agile ceremonies.