DPAI Arena Evaluation Infrastructure Project Maintainer

About JetBrains and DPAI Arena

At JetBrains, code is our passion. We strive to make the strongest, most effective developer tools on earth, automating routine checks and corrections to speed up production and free developers to grow, discover, and create.

We believe in building tools developers love, and the next frontier lies in AI-powered developer tools. The DPAI Arena project aims to define and maintain an open, community-driven benchmark for evaluating AI agents and IDE-embedded AI features at scale. We believe the benchmark industry is only in its early stages, and we’re planning to take part in the evolution of this exciting field.

The Role: DPAI Arena Evaluation Infrastructure Project Maintainer

We’re looking for a DPAI Arena Evaluation Infrastructure Project Maintainer to lead the technical backbone of this initiative – managing the evaluation pipeline, integrating agents, and enabling contributions from the broader community.

As Part Of Our Team, You Will:

• Design, build, and maintain the evaluation infrastructure pipeline with the Eval Infrastructure team, ensuring it supports the goals of DPAI Arena, including scalability, reproducibility, and extensibility.
• Integrate new agents, models, tools, and runtime environments into the pipeline, ensuring compatibility, stability, and maintainability across various configurations.
• Define, document, and maintain contribution guidelines and processes, making it easy and safe for internal teams and external contributors to add new tasks, agents, and evaluation setups.
• Curate the technical process of contributions by reviewing submissions, validating conformity with standards, coordinating merges, and ensuring the consistency, reproducibility, and quality of evaluation artifacts.
• Support the growth of a community-driven ecosystem by enabling contributions, maintaining clear documentation and onboarding flows, engaging with contributors, and ensuring long-term sustainability of the infrastructure.

We’ll be happy to bring you on board if you have:

• Strong experience in building and maintaining evaluation or CI/CD-type infrastructure, pipelines, or related tooling.
• Comfort working with multiple models, runtime environments, tooling setups.
• The ability to create flexible, modular, maintainable infrastructure.
• An understanding of GenAI in the DevTooling domain.
• Proficiency in writing and maintaining clear developer documentation, contribution guidelines, and onboarding processes.
• Good coding and system design skills, including the ability to work with codebases, integrate new components, and manage versioning and configurations.
• Meticulous attention to detail and a quality assurance mindset.
• The ability to coordinate with different teams (product, ML, external contributors) and ensure smooth collaboration.
• A self-motivated and proactive mindset in taking ownership of the long-term technical health of the project.

We’ll Be Especially Thrilled If You Have:

• Experience with AI-agent integration, multi-model pipelines, and benchmarking frameworks.
• Familiarity with open-source and community-driven development workflows, contribution processes, and code review practices.

Key skills/competency

• Evaluation Infrastructure
• CI/CD Pipelines
• AI Agent Integration
• System Design
• Documentation
• Community Management
• Reproducibility
• Scalability
• GenAI
• Quality Assurance