Senior Software Engineer — AI Evaluation & Benchmarks
Alignerr · United States
- Hybrid
- Contract
- $150,000 / year
- United States
Tailored resume — keyword-matched to this role.
Hiring manager — we find who's hiring.
Intro email — drafted to reach them directly.
Job highlights
- Design and build AI model evaluation benchmarks.
- Develop scalable data pipelines for AI evaluation.
- Analyze AI-generated code for errors.
- Test AI reasoning, debugging, and code quality.
- Work remotely on cutting-edge AI projects.
About the role
About The Role
What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software.
This is high-impact, technically demanding work at the intersection of software engineering and AI research. You'll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do.
This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI — this is the role.
Organization:
AlignerrType:
Hourly ContractLocation:
RemoteContract Length:
3 MonthsCommitment:
Full-time availability preferredWhat You'll Do
- Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
- Build and maintain scalable data pipelines for AI evaluation workflows
- Analyze AI-generated code for correctness, reliability, and edge-case failures
- Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
- Work with large code repositories and multi-language environments
- Collaborate on systems that improve how AI models understand and generate software
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to the design of evaluation frameworks that set industry standards
Who You Are
- 4+ years of professional software engineering experience — this is non-negotiable
- Experience working at a high-growth tech company or top-tier software organization
- Expert proficiency in Python — you write clean, performant, well-tested Python code
- Hands-on experience with code repositories and working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and data pipelines
- Track record of working in high-performance engineering environments with large-scale products or platforms
- Strong command of version control systems (Git) and modern development workflows
- Bilingual or native English speaker with strong written communication skills
- Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
Candidates with these additional qualifications have the highest chance of success:
- Senior or Lead-level engineering profiles with a history of technical ownership
- Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience
- Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
- Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
- Background in security engineering or significant open-source contributions
- Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join Us
- Work on cutting-edge AI evaluation projects alongside world-class research teams
- Fully remote — work from anywhere with a reliable internet connection
- Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved
- Freelance autonomy with meaningful, high-stakes engineering work
- Collaborate with a global community of elite engineers and researchers
- Potential for contract extension and ongoing engagement as new evaluation challenges emerge
Key skills/competency
- Senior Software Engineer AI Evaluation
- Python
- AI Model Evaluation
- Data Pipelines
- Coding Benchmarks
- LLM
- Git
- Software Engineering
- Remote Work
- AI Research
Skills & topics
- Senior Software Engineer
- AI Evaluation
- Benchmarks
- Data Pipelines
- Python
- LLM
- Software Development
- Remote
- Contract
- AI Research
How to get hired
- Tailor your resume: Highlight 4+ years of software engineering, Python expertise, and AI/LLM benchmarking experience.
- Showcase your skills: Emphasize Git proficiency, large codebase experience, and high-growth tech company background.
- Demonstrate autonomy: Provide examples of self-directed work and technical rigor in past projects.
- Apply strategically: Clearly articulate how your skills align with evaluating frontier AI models.
- Prepare for technical interviews: Expect deep dives into Python, data pipelines, and AI evaluation methodologies.
Technical preparation
Master Python: focus on clean, performant, tested code.,Practice Git workflows on large codebases.,Build mock data pipelines for AI evaluation.,Develop benchmark tests for coding tasks.
Behavioral questions
Describe a complex codebase you navigated.,How do you ensure code quality autonomously?,Share an experience evaluating AI model performance.,How do you provide technical feedback effectively?
Frequently asked questions
- What is the primary focus of the Senior Software Engineer role at Alignerr?
- The Senior Software Engineer role at Alignerr focuses on designing and building the coding benchmarks and data pipelines essential for evaluating frontier AI models. This involves assessing AI's ability to reason, debug, and generate production-quality software.
- Is this a remote position for the Senior Software Engineer AI Evaluation role?
- Yes, this is a fully remote contract role. Alignerr offers the flexibility to work from anywhere with a reliable internet connection.
- What are the key technical skills required for this Senior Software Engineer position?
- Key technical skills include expert proficiency in Python, hands-on experience with large codebases, strong Git command, and proven experience in designing LLM coding benchmarks and data pipelines.
- What is the expected experience level for the Senior Software Engineer AI Evaluation job?
- A minimum of 4 years of professional software engineering experience is non-negotiable for this role. Experience in high-growth tech companies or top-tier software organizations is also highly valued.
- Does Alignerr prefer candidates with additional programming language skills for this role?
- While Python expertise is essential, proficiency in additional languages such as JavaScript, Go, C++, or others is considered a plus and increases a candidate's chances of success.
- What is the contract length and commitment for this Senior Software Engineer role?
- This is an hourly contract role with an initial length of 3 months. Full-time availability is preferred.
- How does this role contribute to the advancement of AI?
- Your work directly shapes the trajectory of AI by creating evaluation systems that push the boundaries of AI capabilities. The benchmarks you develop will influence how advanced AI systems are measured and improved globally.
- What kind of work environment can I expect as a Senior Software Engineer at Alignerr?
- You can expect a fast-paced engineering environment focused on technically demanding work at the intersection of software engineering and AI research. You'll collaborate with world-class research teams and a global community of elite engineers.