Senior Software Engineer, LLM Evaluation

About the Opportunity at Quik Hire Staffing

Our global AI research client is at the forefront of developing advanced evaluation and benchmarking datasets to significantly improve the performance of large language models (LLMs) in real-world software engineering scenarios. As a Senior Software Engineer, LLM Evaluation, you will play a crucial role in assessing AI-generated code, thereby strengthening model reliability across production-grade engineering workflows.

Role Overview: Senior Software Engineer, LLM Evaluation

This position offers a unique blend of hands-on software engineering expertise with structured AI evaluation and collaborative research. You will contribute to building high-quality datasets essential for training and benchmarking large language models. Working closely with research teams, you will curate complex code examples, provide precise technical solutions, and refine AI-generated outputs across various programming languages.

Key Responsibilities

• Curate and develop realistic software engineering tasks spanning multiple languages, including Python, JavaScript (React), C/C++, Java, Rust, and Go.
• Review, evaluate, and meticulously refine AI-generated code for optimal efficiency, scalability, correctness, and maintainability.
• Collaborate effectively with cross-functional research teams to elevate AI-driven coding solutions against rigorous industry performance benchmarks.
• Design robust verification mechanisms to automatically validate sophisticated software engineering solutions.
• Analyze critical stages of the software development lifecycle (architecture design, API design, prototyping, production deployment, monitoring, and maintenance) and evaluate model performance across these stages.
• Build internal tools or agents designed to detect intricate code quality issues and recurring error patterns.

Requirements

• Several years of professional software engineering experience.
• At least 2 years of continuous full-time experience within a product-focused technology company.
• Strong expertise in building and deploying scalable, production-grade applications.
• Deep understanding of software architecture, debugging methodologies, performance optimization techniques, and established code review standards.
• Proven experience working with modern development workflows and cutting-edge tooling.
• Exceptional written and verbal communication skills, crucial for documenting structured evaluation feedback.

Engagement Details

• Flexible engagement model, requiring a minimum of 10 hours per week, with the potential for up to 40 hours per week.
• Partial overlap with Pacific Time zone hours is required for collaborative efforts.
• This is an independent contractor engagement, without medical or paid leave benefits.
• Initial contract duration is 1 month, with potential for extension contingent on performance and evolving project needs.

Key skills/competency

• LLM Evaluation
• Software Engineering
• AI Research
• Code Review
• Python
• JavaScript
• C/C++
• Java
• Go
• Rust