Python Expert AI Evaluator

@ Braintrust

Project Description

This project involves reviewing AI-generated responses, evaluating technical accuracy and best practices, and providing expert feedback that trains AI systems to write better code. You will work with various programming languages and coding challenges, ensuring AI outputs are functional, efficient, and adhere to industry standards.

Compensation

Compensation is competitive and varies based on education level, institution, and prior experience.

Hiring Profile

• BS, MS, or PhD in Computer Science required.
• Expert in Python with strong skills in algorithms, data structures, and debugging practices.
• Expert-level English proficiency with excellent writing capabilities.
• Experience in fact-checking using public sources and using large language models.
• Strong analytical and structured thinking skills.
• Ability to explain complex programming concepts clearly and accurately.
• Experience with RLHF annotation is preferred.
• Prior software engineering experience is strongly preferred.
• Knowledge of additional programming languages such as C#, Java, SQL, C++, TypeScript, PHP, C, Go, Bash, PowerShell, Rust, or R is a plus.

Application & Onboarding Process

Apply via Braintrust's partner site. You will receive an email with detailed instructions. The process includes a qualification exam (1–2 hours) evaluating role-specific expertise, with compensation on passing, and a project exam (1–2 hours) evaluating adherence to guidelines and ability to produce math-specific deliverables, with compensation upon passing.

Commitment

A minimum of 17–20 hours per week is expected, with a preferred commitment of 8 hours per day based on project scope and timelines.

Key skills/competency

Python, AI evaluation, algorithms, debugging, data structures, coding standards, LLM, RLHF, software engineering, technical feedback