Benchmark Engineer
Coders Connect
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
Benchmark Engineer
What if your job wasn’t just building systems… but proving they’re the best?
We’re partnering with a fast growing open source AI infrastructure company building one of the most exciting technologies in modern search and retrieval systems. Their database powers production grade semantic search, recommendation engines, and large scale RAG pipelines used by teams across the world.
Now they want someone to answer one critical question:“How fast, how accurate, and how reliable are we really?”
And that’s where you come in.
The Mission
This is not a generic backend role.
This is not “run a load test and send a chart.”
You will own how performance is measured, validated, and communicated.
You’ll design realistic benchmarks that reflect real world workloads, not vanity metrics. You’ll explore latency, throughput, recall, memory usage, distributed performance, and cost.
You’ll work directly with core engineers to detect regressions, identify bottlenecks, and shape optimisation strategies.
And when competitors make claims?
You’ll respond with clean data and technical clarity.
What You’ll Be Doing
- Designing reproducible benchmarks for vector search and distributed workloads
- Evaluating performance across latency, throughput, recall, and cost dimensions
- Building benchmarking tooling, automation, CI workflows, dashboards, and reports
- Profiling systems to uncover bottlenecks and regressions
- Running fair, technically rigorous comparisons against alternative solutions
- Turning raw numbers into clear technical narratives for documentation and talks
This role sits at the intersection of engineering depth and technical credibility.
What We’re Looking For
You likely have:
- Strong engineering experience in Rust, Go, Python, or similar
- Solid understanding of databases, distributed systems, or search engines
- Experience with benchmarking, profiling, and performance optimisation
- A deep appreciation for tradeoffs, speed vs accuracy, memory vs latency
- Experience working with large datasets and automated pipelines
- The ability to clearly explain what the numbers actually mean
If you’ve worked with vector search, ANN algorithms, cloud environments, or open source infrastructure, even better.
Who Thrives Here
Someone who enjoys:
- Debating recall vs latency tradeoffs
- Investigating performance regressions
- Designing fair and transparent system comparisons
- Building infrastructure that engineers trust
- Owning a domain end to end
You don’t just want to build features.
You want to measure truth.
Why This Role Is Special
- Work on core infrastructure powering modern AI systems
- Fully remote team with flexible working hours
- Engineering led culture
- High ownership and technical depth
- Real influence on product direction and market positioning
Performance matters more than ever in AI.
This is your chance to define how it’s measured.
If you’re a systems minded engineer who cares about correctness, transparency, and real world performance, let’s talk.
Key skills/competency
- Benchmarking
- Performance Optimization
- Distributed Systems
- AI Infrastructure
- Vector Search
- Profiling
- Automation
- Data Analysis
- Technical Communication
- Open Source
How to Get Hired at Coders Connect
- Research Coders Connect's culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor.
- Tailor your resume for Benchmark Engineer: Highlight experience in Rust, Go, Python, distributed systems, and performance optimization for AI infrastructure roles.
- Showcase benchmarking expertise: Provide concrete examples of designing and implementing performance benchmarks, profiling, and bottleneck identification.
- Prepare for technical deep-dives: Expect questions on vector search, ANN algorithms, cloud environments, and open-source contributions relevant to AI infrastructure.
- Emphasize data-driven communication: Practice explaining complex technical findings clearly, translating raw data into actionable insights for engineers and stakeholders.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background