Senior Software Engineer LLM Evaluation
Elevate Recruitment
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About the Opportunity
One of our global AI research clients is developing advanced evaluation and benchmarking datasets to improve the performance of large language models in real-world software engineering scenarios. This Senior Software Engineer LLM Evaluation role focuses on assessing AI-generated code and strengthening model reliability across production-grade engineering workflows.
Role Overview
As a Senior Software Engineer supporting AI model evaluation, you will contribute to building high-quality datasets used for training and benchmarking large language models. You will work closely with researchers to curate code examples, provide precise technical solutions, and refine AI-generated outputs across multiple programming languages. This role blends hands-on software engineering expertise with structured AI evaluation and research collaboration.
Key Responsibilities
- Curate and develop realistic software engineering tasks across languages such as Python, JavaScript (including React), C/C++, Java, Rust, and Go
- Review, evaluate, and refine AI-generated code for efficiency, scalability, correctness, and maintainability
- Collaborate with cross-functional research teams to enhance AI-driven coding solutions against industry performance benchmarks
- Design verification mechanisms to automatically validate software engineering solutions
- Analyze stages of the software development lifecycle (architecture design, API design, prototyping, production deployment, monitoring, and maintenance) and evaluate model performance across these stages
- Build internal tools or agents to detect code quality issues and error patterns
Requirements
- Several years of professional software engineering experience
- At least 2 years of continuous full-time experience at a product-focused technology company
- Strong expertise in building and deploying scalable, production-grade applications
- Deep understanding of software architecture, debugging, performance optimization, and code review standards
- Experience working with modern development workflows and tooling
- Strong written and verbal communication skills for documenting structured evaluation feedback
Engagement Details
- Flexible engagement: minimum 10 hours per week, up to 40 hours per week
- Partial overlap with Pacific Time required
- Contractor engagement (no medical or paid leave benefits)
- Initial duration: 1 month, with potential extension based on performance and project needs
Key skills/competency
- LLM Evaluation
- AI-generated Code Review
- Software Engineering
- Dataset Curation
- Benchmarking
- Performance Optimization
- Software Architecture
- Debugging
- Multi-language Programming
- Collaboration
How to Get Hired at Elevate Recruitment
- Research Elevate Recruitment's client: Study the global AI research client's mission, values, recent news, and their focus on LLM evaluation.
- Tailor your resume: Highlight deep software engineering, AI model evaluation, and multi-language programming expertise relevant to LLM roles.
- Showcase LLM evaluation projects: Prepare to discuss past work involving AI-generated code review, refinement, and performance benchmarking.
- Emphasize contract work experience: Demonstrate reliability and independence suitable for a flexible hourly contractor engagement.
- Prepare for technical deep-dives: Expect in-depth questions on software architecture, debugging, performance optimization, and modern development workflows.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background