AI QA Engineer (Multilingual)
ChatGPT Jobs · New York, NY
- On site
- Full-time
- $100,000 / year
- New York, NY
Job highlights
- Ensure LLM data quality and accuracy.
- Maintain development environments and test pipelines.
- Analyze data for errors and technical failures.
- Utilize LLMs for cross-lingual dataset tasks.
- Collaborate with engineering on evaluation criteria.
About the role
AI QA Engineer (Multilingual)
Company: Scaled Cognition
Location: New York, NY (Remote)
Key Responsibilities
- Inspect, review, and grade LLM training data, evaluation test cases, and model outputs to ensure quality and accuracy.
- Maintain local development environments, run test pipelines, investigate edge cases, and submit Git/GitHub PRs.
- Analyze training data to identify error cases and technical failures.
- Leverage LLMs for translation, verification, and maintenance of cross-lingual datasets.
- Collaborate with engineering teams to refine evaluation criteria and improve data pipelines.
Key Qualifications
- Strong technical background with hands-on coding experience (Python preferred) and proficiency with Git/GitHub.
- Fluency in English and native or near-native proficiency in at least one other language.
- Deep understanding of Large Language Models (LLMs), failure modes (hallucinations, formatting errors), and prompting techniques.
- Proven experience in Quality Assurance, Data Quality, or Data Engineering, with a track record of auditing large datasets.
- Exceptional written communication skills across multiple languages.
Required Skills & Attributes
- Obsessive attention to detail for finding edge cases and translation errors.
- Ability to handle repetitive data inspection tasks with a "builder" mentality.
- Technical self-sufficiency: comfort with terminal usage, Python scripts, and version control.
- Strong linguistic understanding of nuances required for high-quality cross-lingual evaluation.
- Ability to thrive in a fast-paced, ownership-driven environment.
Key skills/competency
- AI QA Engineer
- LLM
- Python
- Git/GitHub
- Quality Assurance
- Data Quality
- Data Engineering
- Multilingual
- Translation
- Cross-lingual datasets
Skills & topics
- AI QA Engineer
- Quality Assurance
- LLM
- Large Language Models
- Python
- Git
- GitHub
- Data Quality
- Data Engineering
- Multilingual
- Remote
- New York
How to get hired
- Tailor your resume: Highlight Python, Git/GitHub, QA, and multilingual experience.
- Showcase language skills: Emphasize fluency in English and other languages.
- Detail LLM knowledge: Include experience with LLM failure modes and prompting.
- Demonstrate technical aptitude: Mention terminal usage, scripting, and version control.
- Express interest: Write a compelling cover letter detailing your fit.
Technical preparation
Practice Python scripting for data analysis.,Master Git/GitHub for code contribution.,Study LLM failure modes and prompting.,Familiarize with terminal and test pipelines.
Behavioral questions
Describe a time you found a critical data error.,How do you handle repetitive inspection tasks?,How do you approach learning new technical skills?,Share an experience of cross-lingual collaboration.
Frequently asked questions
- What specific languages are prioritized for the AI QA Engineer role at Scaled Cognition?
- While fluency in English is required, Scaled Cognition specifically seeks native or near-native proficiency in at least one other language. The job description does not list specific preferred languages beyond this, so highlighting any additional languages you possess beyond English would be beneficial.
- How important is Python experience for the AI QA Engineer position at Scaled Cognition?
- Python experience is highly preferred for the AI QA Engineer role at Scaled Cognition. The job description mentions "hands-on coding experience (Python preferred)" and that candidates should be comfortable with "Python scripts." Highlighting your Python skills and any relevant projects is recommended.
- What does Scaled Cognition mean by 'remote' for this AI QA Engineer position?
- The AI QA Engineer role at Scaled Cognition is listed as 'Remote (New York, NY)'. This typically means you can work from home, but may be expected to be available for occasional in-person meetings or have a primary work location within the specified state or region.
- How can I demonstrate my 'obsessive attention to detail' for the AI QA Engineer job?
- To demonstrate obsessive attention to detail for the AI QA Engineer role, highlight specific examples in your resume or cover letter where you identified subtle errors, edge cases, or quality issues in data or systems. Mentioning experience with meticulous auditing or linguistic nuance will also be effective.
- What kind of 'ownership-driven environment' can I expect at Scaled Cognition?
- An ownership-driven environment means Scaled Cognition likely empowers its employees to take initiative and responsibility for their work. For the AI QA Engineer role, this might involve taking full ownership of testing pipelines, data quality initiatives, or specific evaluation criteria without constant direct supervision.