6 days ago
LLM Data Quality & Tooling Specialist
Keystone Recruitment
Hybrid
Contractor
$120,000
Hybrid
Job Overview
Job TitleLLM Data Quality & Tooling Specialist
Job TypeContractor
CategoryCommerce
Experience5 Years
DegreeMaster
Offered Salary$120,000
LocationHybrid
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
LLM Data Quality & Tooling Specialist at Keystone Recruitment
A global AI-focused organization, Keystone Recruitment, is seeking an LLM Data Quality & Tooling Specialist to support large language model training initiatives through structured data validation and lightweight tooling development. This remote role blends hands-on quality control with practical tool-building to improve dataset reliability, clarity, and evaluation standards across STEM and humanities domains.
You will work closely with internal experts and AI teams to ensure training data meets rigorous quality standards while building simple, effective workflows to scale review processes.
Key Responsibilities
- Perform quality checks (QC) on generated datasets including Q&A pairs, rubric-based datasets, evaluation samples, and agentic workflows
- Review content across both STEM and humanities domains for logical consistency, accuracy, clarity, and completeness
- Develop clear, actionable QC guidelines to help domain experts independently review their samples
- Design and build lightweight Python-based utilities (primarily in Google Colab) to streamline validation workflows
- Translate quality requirements into structured review checklists, scripts, and semi-automated tools
- Identify recurring data issues and recommend process improvements
- Communicate findings clearly and professionally to cross-functional stakeholders
Required Qualifications
- Strong analytical thinking and attention to detail
- Experience performing structured content validation or quality assurance
- Proficiency in Python (especially for lightweight scripting and automation)
- Comfortable working in Google Colab or similar notebook environments
- Ability to convert ambiguous QC requirements into practical workflows and tools
- Excellent written communication skills
- Ability to work full-time with required PST overlap
Engagement Details
- Contractor assignment (no medical/paid leave)
- Full-time commitment with 4 hours of overlap with PST
- Duration: 10 months
- Fully remote
- Potential for contract extension based on performance
Evaluation Process
- Job interest form
- Interview
- Selection and onboarding
Key skills/competency
- Data Quality
- LLM Training
- Python Scripting
- Google Colab
- Content Validation
- Quality Assurance
- Analytical Thinking
- Process Improvement
- Cross-functional Communication
- Dataset Evaluation
How to Get Hired at Keystone Recruitment
- Research Keystone Recruitment's culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor.
- Tailor your resume: Highlight your LLM data quality, Python scripting, and content validation experience to align with the LLM Data Quality & Tooling Specialist role.
- Showcase Python/Colab expertise: Provide examples of lightweight tools or scripts you've developed for data automation or quality control, emphasizing practical application.
- Prepare for technical and behavioral interviews: Be ready to discuss your analytical thinking, attention to detail, and problem-solving skills in data quality scenarios.
- Demonstrate communication skills: Practice articulating complex data quality issues and process improvements clearly and professionally for cross-functional teams.
Frequently Asked Questions
Find answers to common questions about this job opportunity
01What is the primary focus of the LLM Data Quality & Tooling Specialist role at Keystone Recruitment?
02Is the LLM Data Quality & Tooling Specialist position a full-time employee role?
03What technical skills are essential for an LLM Data Quality & Tooling Specialist at Keystone Recruitment?
04How does the LLM Data Quality & Tooling Specialist contribute to AI model development?
05What kind of content will the LLM Data Quality & Tooling Specialist be reviewing?
Explore similar opportunities that match your background