LLM Data Quality Specialist
Keystone Recruitment
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
Role Overview: LLM Data Quality Specialist
A global AI-focused organization, Keystone Recruitment, is seeking an LLM Data Quality Specialist to support large language model training initiatives through structured data validation and lightweight tooling development. This role blends hands-on quality control with practical tool-building to improve dataset reliability, clarity, and evaluation standards across STEM and humanities domains.
You will work closely with internal experts and AI teams to ensure training data meets rigorous quality standards while building simple, effective workflows to scale review processes.
Key Responsibilities
- Perform quality checks (QC) on generated datasets including Q&A pairs, rubric-based datasets, evaluation samples, and agentic workflows.
- Review content across both STEM and humanities domains for logical consistency, accuracy, clarity, and completeness.
- Develop clear, actionable QC guidelines to help domain experts independently review their samples.
- Design and build lightweight Python-based utilities (primarily in Google Colab) to streamline validation workflows.
- Translate quality requirements into structured review checklists, scripts, and semi-automated tools.
- Identify recurring data issues and recommend process improvements.
- Communicate findings clearly and professionally to cross-functional stakeholders.
Required Qualifications
- Strong analytical thinking and attention to detail.
- Experience performing structured content validation or quality assurance.
- Proficiency in Python (especially for lightweight scripting and automation).
- Comfortable working in Google Colab or similar notebook environments.
- Ability to convert ambiguous QC requirements into practical workflows and tools.
- Excellent written communication skills.
- Ability to work full-time with required PST overlap.
Engagement Details
This is an hourly contract assignment (independent contractor) with no medical or paid leave. A full-time commitment is required with 4 hours of overlap with PST. The duration is 10 months, fully remote, with potential for contract extension based on performance.
Evaluation Process
- Job interest form
- Interview
- Selection and onboarding
Key skills/competency
- LLM Data Quality
- Python Scripting
- Google Colab
- Data Validation
- Quality Assurance
- AI Training Data
- Analytical Thinking
- Content Review
- Workflow Automation
- Cross-functional Communication
How to Get Hired at Keystone Recruitment
- Research Keystone Recruitment's culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor.
- Tailor your resume for LLM Data Quality Specialist: Highlight Python, data validation, Google Colab, and quality assurance skills for AI training data.
- Showcase practical tooling experience: Provide concrete examples of lightweight Python scripts or tools you've built to automate quality control processes.
- Prepare for technical and behavioral interviews: Expect questions on data quality methodologies, Python scripting challenges, and problem-solving scenarios related to LLMs.
- Demonstrate strong communication skills: Emphasize your ability to translate complex technical requirements into clear guidelines for diverse stakeholders.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background