Software Engineer

Software Engineer (AI Systems Evaluator)

Keystone Recruitment's client, a leading AI research organization, is seeking a Software Engineer to evaluate and improve advanced conversational AI systems. This role focuses on enhancing how large language models (LLMs) reason about code, generate solutions, and explain technical concepts across various programming and system design scenarios. This is an hourly contract position for independent contractors.

Key Responsibilities

• Evaluate AI-generated responses to software engineering and coding queries for correctness, clarity, and completeness.
• Execute and test code to validate functionality, performance, and edge-case handling.
• Perform fact-checking using authoritative technical references and public sources.
• Annotate model outputs by identifying strengths, weaknesses, bugs, and conceptual gaps.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Ensure outputs align with established conversational and technical guidelines.
• Apply standardized evaluation rubrics and benchmarks consistently.

Required Qualifications

• Bachelor’s, Master’s, or PhD in Computer Science or a closely related field.
• Significant professional experience in software engineering or system design.
• Expert-level proficiency in at least one major programming language (e.g., Python, Java, C++, JavaScript, Go, Rust).
• Ability to independently solve medium-to-hard algorithmic problems.
• Experience contributing to open-source projects with accepted pull requests.
• Strong familiarity with using LLMs for coding and understanding their limitations.
• Exceptional attention to detail and ability to detect subtle technical errors.

Preferred Qualifications

• Prior experience with RLHF, model evaluation, or technical data annotation.
• Background in competitive programming or algorithmic problem solving.
• Experience reviewing or maintaining production-level code.
• Familiarity with multiple programming paradigms and technology stacks.
• Ability to explain complex technical topics to non-technical audiences.

What Success Looks Like

• You consistently identify logical errors, inefficiencies, and misleading explanations in AI-generated code.
• Your feedback measurably improves the accuracy, reliability, and clarity of model outputs.
• You deliver high-quality, reproducible evaluation artifacts that strengthen AI system performance.

Contract & Payment Terms

• Independent contractor engagement.
• Fully remote with flexible scheduling.
• Weekly payments via Stripe or Wise.
• Project scope and duration may vary based on performance and client needs.
• No access to confidential or proprietary employer data is required.
• H1-B and STEM OPT sponsorship is not available.

Key skills/competency

• Large Language Models (LLMs)
• Software Engineering
• Code Evaluation
• Algorithmic Problem Solving
• Python (or Java, C++, JavaScript, Go, Rust)
• System Design
• Technical Fact-Checking
• AI Model Annotation
• Open-Source Contributions
• Debugging & Testing