Software Engineer, Systems Design

Software Engineer, Systems Design at Keystone Recruitment

Keystone Recruitment is seeking a Software Engineer, Systems Design to join one of its clients, a leading AI research organization. This hourly contract role focuses on evaluating and enhancing advanced conversational AI systems designed to assist users with real-world software engineering and coding tasks. The position requires a deep understanding of how Large Language Models (LLMs) reason about code, generate solutions, and explain technical concepts across diverse programming and system design scenarios.

Key Responsibilities

• Evaluate AI-generated responses to software engineering and coding queries for correctness, clarity, and completeness.
• Execute and test code to validate functionality, performance, and edge-case handling.
• Perform fact-checking using authoritative technical references and public sources.
• Annotate model outputs by identifying strengths, weaknesses, bugs, and conceptual gaps.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Ensure outputs align with established conversational and technical guidelines.
• Apply standardized evaluation rubrics and benchmarks consistently.

Required Qualifications

• Bachelor’s, Master’s, or PhD in Computer Science or a closely related field.
• Significant professional experience in software engineering or system design.
• Expert-level proficiency in at least one major programming language (e.g., Python, Java, C++, JavaScript, Go, Rust).
• Ability to independently solve medium-to-hard algorithmic problems.
• Experience contributing to open-source projects with accepted pull requests.
• Strong familiarity with using LLMs for coding and understanding their limitations.
• Exceptional attention to detail and ability to detect subtle technical errors.

Preferred Qualifications

• Prior experience with RLHF, model evaluation, or technical data annotation.
• Background in competitive programming or algorithmic problem solving.
• Experience reviewing or maintaining production-level code.
• Familiarity with multiple programming paradigms and technology stacks.
• Ability to explain complex technical topics to non-technical audiences.

What Success Looks Like

• You consistently identify logical errors, inefficiencies, and misleading explanations in AI-generated code.
• Your feedback measurably improves the accuracy, reliability, and clarity of model outputs.
• You deliver high-quality, reproducible evaluation artifacts that strengthen AI system performance.

Contract & Payment Terms

• Independent contractor engagement.
• Fully remote with flexible scheduling.
• Weekly payments via Stripe or Wise.
• Project scope and duration may vary based on performance and client needs.
• No access to confidential or proprietary employer data is required.
• H1-B and STEM OPT sponsorship is not available.

Application Process

• Submit your resume for review.
• Selected candidates will complete a short technical and evaluation assessment.

Key skills/competency

• Software Engineering
• Systems Design
• Large Language Models (LLMs)
• Code Evaluation
• Algorithmic Problem Solving
• Technical Annotation
• Python/Java/C++/Go/Rust
• Debugging
• AI System Performance
• Open-Source Contribution