AI Quality Analyst
Texas Sports Academy · United States
- Hybrid
- Full-time
- $60,000 / year
- United States
Tailored resume — keyword-matched to this role.
Hiring manager — we find who's hiring.
Intro email — drafted to reach them directly.
Job highlights
- Analyze AI chatbot conversations for quality issues.
- Create and maintain test files and knowledge base.
- Fix issues and prevent regressions in AI systems.
- Work with JSON, Markdown, and terminal commands.
- Requires native English and high attention to detail.
About the role
AI Quality Analyst
We run an AI-powered SMS bot and an AI-graded call training system for a private school network. Both systems have extensive test suites (400+ test files) and a RAG-based knowledge base that drives bot behavior.
Own The Full Quality Cycle
- Review production conversations to find bad bot responses.
- Create test files (JSON) that reproduce the issue.
- Fix the knowledge base content (Markdown files, WRONG/RIGHT examples).
- Run test suites to verify fixes and catch regressions.
- Maintain and expand the knowledge base as our program evolves.
This is NOT a software engineering role. The infrastructure is built. You're operating it — editing JSON, writing Markdown, running CLI scripts, reading test reports.
What You'll Work With
- JSON test files (you'll write and edit these daily)
- Markdown knowledge base documents
- Terminal commands (copy-paste and run scripts)
- Git (commit, push, basic branching)
- VS Code or similar editor
Requirements
- Native-level English fluency (non-negotiable). You'll be writing realistic SMS conversations between parents and our school. The language has to sound like a real person texting, not a corporate chatbot. You'll also be writing precise pass/fail evaluation criteria.
- Comfortable editing JSON and Markdown in an IDE.
- Can run commands in a terminal without hand-holding.
- Extreme attention to detail.
- Ability to learn a complex domain quickly (education, state government programs, etc).
Nice-to-Haves
- Experience writing test cases or QA documentation.
- Experience with chatbot QA, conversational AI testing, or LLM evaluation.
Hours & Ramp-Up
- 30 hrs/week to start, ramping up as needed. US time zones required.
- The first 1-2 weeks will be focused on learning our domain — how our school works, how the state voucher program works, compliance rules, etc. We have extensive internal documentation.
Key skills/competency
- AI Quality Analyst
- Conversational AI
- LLM Evaluation
- Chatbot QA
- JSON
- Markdown
- Test Case Creation
- Regression Testing
- Attention to Detail
- Domain Knowledge
Skills & topics
- AI Quality Analyst
- Conversational AI
- LLM Evaluation
- Chatbot QA
- JSON
- Markdown
- Test Case Creation
- Regression Testing
- Attention to Detail
- Domain Knowledge
- Education Technology
- SMS Bot
- RAG
- US Time Zones
How to get hired
- Tailor your resume: Highlight experience with JSON, Markdown, QA, and conversational AI testing.
- Showcase attention to detail: Provide examples of meticulous work in previous roles.
- Demonstrate domain adaptability: Emphasize your ability to quickly learn complex subjects like education.
- Prepare for technical tasks: Be ready to discuss editing JSON/Markdown and using terminal commands.
Technical preparation
Practice writing realistic SMS conversations.,Create JSON files for bug reproduction.,Edit Markdown for knowledge base updates.,Simulate running CLI scripts and Git commands.
Behavioral questions
Describe a time you found a subtle error.,How do you learn complex new domains quickly?,Explain how you ensure language sounds natural.,How do you manage tasks with extreme detail?
Frequently asked questions
- What is the work arrangement for the AI Quality Analyst role at Texas Sports Academy?
- This is a remote position requiring availability during US time zones. While the core work is remote, the company culture emphasizes collaboration and efficiency, so prompt communication is key.
- What technical skills are essential for the AI Quality Analyst job?
- Essential technical skills include proficiency in editing JSON and Markdown files within an IDE, comfort with terminal commands for running scripts, and basic Git usage for version control. Familiarity with VS Code or similar editors is also expected.
- Does Texas Sports Academy offer opportunities for growth in this AI Quality Analyst role?
- While this role focuses on operating existing infrastructure, there is potential for growth as the program evolves. Expanding and maintaining the knowledge base offers opportunities to deepen expertise in conversational AI and LLM evaluation.
- What is the expected time commitment for the AI Quality Analyst position?
- The role starts at approximately 30 hours per week, with the potential to ramp up based on evolving needs. This flexible structure allows for a focus on quality and learning.
- How does Texas Sports Academy ensure the quality of its AI systems?
- Texas Sports Academy employs a rigorous quality cycle, including reviewing production conversations, creating detailed test files, fixing knowledge base content, and running extensive test suites to catch regressions.
- What makes the English fluency requirement non-negotiable for this AI Quality Analyst role?
- Native-level English fluency is crucial because the role involves writing realistic SMS conversations between parents and school staff, as well as precise pass/fail evaluation criteria. The language must sound natural and human-like.
- What kind of domain knowledge is expected for the AI Quality Analyst position?
- The role requires the ability to quickly learn complex domains, specifically related to the education sector, state government programs, and compliance rules relevant to the private school network served by the AI systems.