AI Quality Assurance Analyst
Allied Benefit Systems
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
Position Summary: AI Quality Assurance Analyst
As the AI Quality Assurance Analyst, you will serve as the quality champion for Allied Benefit Systems' emerging AI capabilities. Your primary responsibility will be to ensure that every AI model, integration, and automation functions reliably and safely within core business processes. This role involves close collaboration with developers, product owners, and business users to design robust tests that push the limits of applications and deliver trustworthy AI solutions.
Essential Functions
- Test AI features integrated into critical systems such as Dynamics CRM, QicLink claims platform, and Docuvantage, ensuring correct functionality in end-to-end workflows.
- Write and automate test cases for AI outputs and traditional software features, defining clear pass/fail criteria, using statistical sampling where appropriate, to validate AI results.
- Take ownership of regression testing for AI capabilities, continuously re-running critical test scenarios as models are updated or data changes to proactively identify issues.
- Contribute to the overall test strategy and perform adversarial or edge-case testing, including stress tests and prompt injections, to evaluate the robustness and security of AI components.
- Document and recommend solutions for identified issues, coordinating with the AI Product Owner and Subject Matter Experts (SMEs) to clarify requirements for fixes and improvements.
- Collaborate closely with developers and the Product Owner to report bugs, verify fixes, and enhance requirements, emphasizing shared accountability for quality across the team.
Education
A Bachelor’s degree in a related field or equivalent work experience is required.
Experience And Skills
- 7+ years of experience in software Quality Assurance/testing roles, with significant experience in enterprise systems or complex integrations.
- Proven expertise in test automation (UI and API) and building regression test suites within CI/CD environments.
- Familiarity with AI/ML system testing principles, including comfort in validating statistically varying outputs and determining acceptable accuracy thresholds.
- Strong analytical skills for identifying edge cases and pinpointing defects across interconnected systems (frontend, backend, data, model).
- Excellent documentation and communication skills, vital for writing test plans, reporting issues, and guiding teams on quality best practices.
- Experience in industries such as insurance or healthcare is a plus, providing valuable domain context for Allied Benefit Systems' processes.
- Proficiency with test automation tools like Selenium or Cypress for web interface testing, and Postman or REST-assured for API testing.
- Experience with testing frameworks such as PyTest or JUnit, integrated with CI/CD pipelines (e.g., Azure DevOps, Jenkins) for continuous testing.
- Competence in version control (Git) for maintaining test scripts, and using monitoring/logging tools (e.g., Azure Monitor, Splunk) for issue investigation.
- Familiarity with performance testing or security testing tools for evaluating AI systems under load or threat scenarios.
Position Competencies
- Job Knowledge
- Time Management
- Accountability
- Communication
- Initiative
- Customer Focus
Work Environment
This is a remote position, emphasizing flexibility and the comfort of working from home. Candidates must have a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload to support a smooth and efficient remote work experience.
Compensation & Benefits
Allied Benefit Systems offers a competitive Total Rewards package, including medical, dental, vision, life & disability insurance, generous paid time off, tuition reimbursement, EAP, and a technology stipend. Compensation is contingent upon meeting qualifications and successful completion of the interview process.
Key skills/competency
- AI Quality Assurance
- Test Automation
- CI/CD
- Regression Testing
- AI/ML System Testing
- Enterprise Systems
- API Testing
- UI Testing
- Edge Case Testing
- Data Validation
How to Get Hired at Allied Benefit Systems
- Research Allied Benefit Systems' culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor, focusing on their remote-first approach and commitment to innovation.
- Tailor your resume for AI Quality Assurance Analyst: Highlight your 7+ years of QA experience, expertise in test automation (UI/API), and any exposure to AI/ML system testing, explicitly mentioning tools like Selenium, Cypress, PyTest, or JUnit.
- Showcase AI/ML testing understanding: Prepare to discuss your approach to validating statistically varying AI outputs, determining accuracy thresholds, and performing adversarial testing during your Allied Benefit Systems interview.
- Emphasize remote work readiness: Be ready to articulate how you thrive in a remote environment, your communication strategies, and your understanding of the internet speed requirements for the AI Quality Assurance Analyst role.
- Prepare for competency-based questions: Allied Benefit Systems values Job Knowledge, Accountability, and Communication; frame your experiences to demonstrate these competencies, particularly in cross-functional team settings.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background