AI Agent Testing Specialist
Braintrust
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About AI Agent Testing Specialist
At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.
What We Do
The Mindrift platform connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.
About The Role
You will design realistic and structured evaluation scenarios for LLM-based agents. Responsibilities include creating test cases that simulate human-performed tasks, defining gold-standard behavior, and ensuring tests are clear, well-scored, and reusable.
- Design structured test scenarios based on real-world tasks.
- Define expected agent behaviors and acceptable performance.
- Annotate task steps, expected outputs, and edge cases.
- Collaborate with developers to improve testing clarity.
- Review and adapt tests based on agent outcomes.
How To Get Started
Simply apply by submitting your resume in English and indicate your level of English. Qualify and contribute to projects on your own schedule, influencing the future of AI while working remotely.
Requirements
- Bachelor's and/or Master’s Degree in a related field.
- Background in QA, software testing, data analysis, or NLP annotation.
- Good understanding of test design principles, reproducibility, and coverage.
- Strong written English communication skills.
- Experience with JSON/YAML, Python, and JS basics.
- Curiosity and willingness to work with AI-generated content and agent logs.
Nice to Have
Experience in writing manual or automated test cases, familiarity with LLM capabilities and failure modes, and understanding of scoring metrics like precision, recall, and coverage.
Benefits
- Flexible, remote, freelance project.
- Opportunity to work on advanced AI projects.
- Enhance your portfolio with cutting-edge AI work.
- Work on your own schedule from anywhere in the world.
Key skills/competency
- AI
- Testing
- QA
- LLM
- Evaluation
- Automation
- Python
- JavaScript
- NLP
- Data Analysis
How to Get Hired at Braintrust
- Customize Resume: Highlight relevant IT and QA experience.
- Research Braintrust: Understand its projects and culture.
- Follow Guidelines: Tailor your application to job requirements.
- Prepare Examples: Share past testing and scenario work.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background