AI Evaluation Engineer
People Prime Worldwide · India
- Hybrid
- Contract
- $80,000 / year
- India
Job highlights
- Evaluate AI coding outputs and identify differences.
- Requires 5+ years of software engineering experience.
- Must be proficient with terminal and coding tools.
- Familiarity with agentic coding environments is key.
- Requires strong quality judgment and process adherence.
About the role
About Company
Our client is a Palo Alto-based AI infrastructure and talent platform founded in 2018. It helps companies connect with remote software developers using AI-powered vetting and matching technology. Originally branded as the “Intelligent Talent Cloud,” enabled companies to “spin up their engineering dream team in the cloud” by sourcing and managing vetted global talent. In recent years, they have evolved to support AI infrastructure and AGI workflows, offering services in model training, fine-tuning, and deployment—powered by their internal AI platform, ALAN, and backed by a vast talent network. They reported $300 million in revenue and reached profitability. Their growth is driven by demand for annotated training data from AI labs, including major clients like OpenAI, Google, Anthropic, and Meta.
Job Description
Job Title: Agentic Coding Annotator - Online / Offline Tasks
Location: Pan India
Experience: 5+ yrs.
Employment Type: Contract to hire
Work Mode: Remote
Requirements
- Software Engineering Fluency (Mandatory): 5+ years of experience in software engineering, QA, developer tooling, data/ML engineering, or similar code-heavy roles.
- Strong hands-on experience in at least 1–2 programming languages or ecosystems. Representative languages include: Python, JavaScript/TypeScript, Rust, Java, C/C++, Bash/CLI environments, Haskell, Swift, SQL, or other production-relevant ecosystems.
- Ability to: Read and understand unfamiliar codebases, Run and interpret tests, scripts, and CLI tools, Debug issues and reason about edge cases or partial fixes, Evaluate whether an implementation is functionally correct.
- Terminal & Tooling Skills (Mandatory): Comfortable working in Linux/Ubuntu-like environments. Proficient with: Terminal workflows, Git basics, Code editors or IDEs, Package managers and test runners, JSON, YAML, and Markdown. Familiarity with Docker and reproducible environments is a strong plus, especially for offline work.
- Coding-Agent Workflow Familiarity (Mandatory): Comfortable working with or quickly adapting to agentic coding environments, such as: OpenCode, Claude Code, Cursor, Similar coding-agent tools.
- Quality Judgment & Annotation Accuracy (Mandatory): Ability to compare multiple model trajectories and identify meaningful differences, Distinguish correctness from style, communication quality, and agent behavior, Evaluate solutions consistently using defined rubrics, Follow detailed process instructions without deviation, Maintain consistency across repeated or similar evaluations, Write concise, evidence-based rationales (not generic summaries).
Work Style
- Highly detail-oriented and process-driven.
- Comfortable with repetitive, high-precision evaluation work.
- Able to maintain consistency across long tasks and multiple model runs.
- Proactively flags ambiguity instead of making assumptions.
- Balances realism with strict evaluation consistency.
Additional Preferred Qualifications (Offline / Senior Candidates)
- Strong Docker skills and experience building/debugging reproducible environments.
- Experience working in large, complex repositories (not just small or greenfield projects).
- Demonstrated originality and sound engineering judgment in defining technical problems.
- Ability to design realistic, non-trivial tasks that go beyond tutorials, README flows, or simple bug fixes.
Key skills/competency
- AI Coding Annotator
- Software Engineering
- Python
- JavaScript
- Rust
- Linux
- Git
- Docker
- Agentic Coding
- Quality Evaluation
Skills & topics
- AI Coding Annotator
- Software Engineering
- Python
- JavaScript
- Rust
- Linux
- Git
- Docker
- Agentic Coding
- Quality Evaluation
- Data Annotation
- ML Engineering
- QA
- Developer Tooling
- Remote
- Contract to Hire
How to get hired
- Tailor your resume: Highlight 5+ years of software engineering, QA, or ML engineering experience. Emphasize your proficiency in at least 1-2 programming languages (Python, JavaScript, Rust, Java, C/C++, etc.) and your experience with terminal workflows, Git, and code editors.
- Showcase technical skills: Clearly list your experience with Linux/Ubuntu, package managers, test runners, JSON, YAML, and Markdown. If you have Docker experience, make sure to mention it.
- Demonstrate AI workflow familiarity: Detail any experience you have with agentic coding environments like OpenCode, Claude Code, or Cursor. If you don't have direct experience, express your eagerness to adapt quickly.
- Emphasize quality and process: Highlight your detail-oriented, process-driven work style. Provide examples of how you maintain consistency, follow instructions meticulously, and evaluate solutions rigorously using rubrics.
- Prepare for the interview: Be ready to discuss your problem-solving approach, how you debug code, and your ability to understand unfamiliar codebases. If applying for a senior role, prepare to discuss designing realistic technical tasks.
Technical preparation
Master Python, JavaScript, or Rust for coding tasks.,Practice Linux terminal commands and Git workflows.,Familiarize yourself with Docker and reproducible environments.,Study agentic coding tools like Cursor or OpenCode.
Behavioral questions
Describe a complex codebase you analyzed.,How do you ensure consistency in repetitive tasks?,How would you evaluate AI code for correctness?,How do you handle ambiguity in instructions?
Frequently asked questions
- What is the primary responsibility of an AI Coding Annotator at People Prime Worldwide?
- As an AI Coding Annotator, your main role is to evaluate and annotate code generated by AI models. This involves comparing different AI outputs, assessing their functional correctness, and providing detailed rationales based on defined rubrics for tasks related to AI infrastructure and AGI workflows.
- What technical skills are mandatory for the AI Coding Annotator role?
- Mandatory technical skills include strong software engineering fluency with 5+ years of experience, hands-on coding in at least one production-relevant language (e.g., Python, JavaScript, Rust, Java), proficiency in Linux/Ubuntu environments, Git, code editors, package managers, test runners, and experience with JSON, YAML, and Markdown. Familiarity with agentic coding tools is also required.
- Does this AI Coding Annotator position offer remote work?
- Yes, this AI Coding Annotator position is a remote role, allowing you to work from anywhere in India. The company focuses on connecting global talent with AI infrastructure needs.
- What kind of programming languages and tools are relevant for this AI Coding Annotator job?
- The job requires strong familiarity with production-relevant programming languages such as Python, JavaScript/TypeScript, Rust, Java, C/C++, Bash/CLI, Haskell, Swift, and SQL. You'll also use terminal workflows, Git, code editors, package managers, test runners, and tools like JSON, YAML, and Markdown. Docker experience is a significant plus.
- How important is attention to detail and consistency for an AI Coding Annotator?
- Attention to detail and consistency are paramount. The role requires a highly detail-oriented and process-driven approach to ensure accurate and reliable evaluation of AI model outputs. You must be able to maintain consistency across repetitive tasks and long evaluation sessions.
- What are the preferred qualifications for senior candidates or offline work in this role?
- For senior candidates or offline work, strong Docker skills for building and debugging reproducible environments are highly preferred. Experience with large, complex code repositories and the ability to design realistic, non-trivial technical tasks that go beyond basic tutorials are also valued.
- What is the employment type for the AI Coding Annotator position?
- The employment type for this position is contract-to-hire, offering a path to a permanent role after an initial contract period.
- What is the typical notice period expected for an AI Coding Annotator candidate?
- The job description indicates a preference for immediate joiners, suggesting that candidates should be able to start promptly after the hiring process is complete.