Site Reliability Engineer, Inference Infrastructure
Cohere · New York, NY
- On site
- Full-time
- $150,000 / year
- New York, NY
Email the hiring manager to get a response.
Get their verified email + an intro that's ready to send.
Subject: Interested in the Site Reliability Engineer, Inference Infrastructure role at Cohere
Hi Morgan — I came across the Site Reliability Engineer, Inference Infrastructure opening and wanted to reach out directly. I've spent the last few years doing exactly this kind of work, and Cohere stood out because…
✎ Personalized to your résumé after sign-up.
- ✓ Verified email of the hiring manager
- ✓ Intro email personalized to your résumé
- ✓ $9/mo = unlimited — any job link
Secure checkout · cancel anytime
Job highlights
- Build and operate AI inference infrastructure.
- Automate systems for deployment and observability.
- Ensure high availability and meet SLOs.
- Collaborate with developers and influence roadmaps.
- Work with cutting-edge large language models.
About the role
About Cohere
Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems. We’re training and deploying frontier models for enterprises who are building AI systems. We believe that our work is instrumental to the widespread adoption of AI and we are looking for folks that want to be part of that. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. Cohere is a team of researchers, engineers, designers, and more, who are all passionate about their craft. We are a global technology company co-headquartered in Toronto and San Francisco, with key offices in London, New York City, Montreal, Seoul, Germany and Paris. Join us!
Why this role?
Are you energized by building high-performance, scalable and reliable machine learning systems? Do you want to help define and build the next generation of AI platforms powering advanced NLP applications? We are looking for a Site Reliability Engineer to join the Model Serving team at Cohere. The team is responsible for developing, deploying, and operating the AI platform delivering Cohere's large language models through easy to use API endpoints. In this role, you will work closely with many teams to deploy optimized NLP models to production in low latency, high throughput, and high availability environments. You will also get the opportunity to interface with customers and create customized deployments to meet their specific needs.
As a Site Reliability Engineer You Will
- Build self-service systems that automate managing, deploying and operating services. This includes our custom Kubernetes operators that support language model deployments.
- Automate environment observability and resilience. Enable all developers to troubleshoot and resolve problems.
- Take steps required to ensure we hit defined SLOs, including participation in an on-call rotation.
- Build strong relationships with internal developers and influence the Infrastructure team’s roadmap based on their feedback.
- Develop our team through knowledge sharing and an active review process.
You May Be a Good Fit If You Have
- 5+ years of engineering experience running production infrastructure at a large scale
- Experience designing large, highly available distributed systems with Kubernetes, and GPU workloads on those clusters
- Experience with Kubernetes dev and production coding and support
- Experience with GCP, Azure, AWS, OCI, multi-cloud on-prem / hybrid serving
- Experience in designing, deploying, supporting, and troubleshooting in complex Linux-based computing environments
- Experience in compute/storage/network resource and cost management
- Excellent collaboration and troubleshooting skills to build mission-critical systems, and ensure smooth operations and efficient teamwork
- The grit and adaptability to solve complex technical challenges that evolve day to day
- Familiarity with computational characteristics of accelerators (GPUs, TPUs, and/or custom accelerators), especially how they influence latency and throughput of inference.
- Strong understanding or working experience with distributed systems.
- Experience in Golang, C++ or other languages designed for high-performance scalable servers).
How And Where We Work
Cohere is remote-friendly. We have offices in Toronto, San Francisco, New York City, London, Paris, Montreal, and more coming soon. For those in the office: a daily lunch program, plenty of snacks, and regular community and social events. For those not near an office: a co-working benefit so you can work alongside others in your city.
If any of the above doesn’t line up exactly with your experience, we still encourage you to apply. We strive to create an inclusive work environment for all; we welcome applicants from all backgrounds and are committed to providing equal opportunities. Should you require any accommodations during the recruitment process, please submit an Accommodations Request Form, and we will work together to meet your needs. We may use AI-enabled tools to screen and assess applicants against the criteria for this position. This helps our recruiters identify potentially qualified candidates, but it doesn't limit the applications our recruiters may review or consider.
Key skills/competency
- Site Reliability Engineering
- Inference Infrastructure
- Kubernetes
- Distributed Systems
- Cloud Computing (GCP, Azure, AWS)
- Observability
- High Availability
- Performance Optimization
- Golang/C++
- Machine Learning Infrastructure
Skills & topics
- Site Reliability Engineer
- Inference Infrastructure
- AI
- Machine Learning
- Kubernetes
- Distributed Systems
- Cloud Computing
- GCP
- Azure
- AWS
- Golang
- C++
- Observability
- High Availability
- Production Infrastructure
How to get hired
- Tailor your resume: Highlight your 5+ years of production infrastructure experience, Kubernetes, and distributed systems expertise for this Site Reliability Engineer role.
- Showcase cloud skills: Emphasize your experience with GCP, Azure, AWS, or multi-cloud environments, and managing GPU workloads.
- Quantify achievements: Provide examples of how you've improved system availability, reduced latency, or managed costs in complex Linux environments.
- Demonstrate collaboration: Use your resume to show strong troubleshooting and communication skills, essential for working with internal developers.
- Prepare for technical interviews: Be ready to discuss distributed systems, Kubernetes operators, and the computational characteristics of accelerators.
Technical preparation
Behavioral questions
Frequently asked questions
- What is the typical career progression for a Site Reliability Engineer at Cohere?
- While specific paths can vary, a Site Reliability Engineer at Cohere can expect opportunities to grow into more senior technical roles, specialize in specific areas of infrastructure or AI deployment, or potentially move into leadership positions within the engineering team. Your growth will be supported through knowledge sharing and active review processes.
- What kind of AI models will I be working with as a Site Reliability Engineer at Cohere?
- As a Site Reliability Engineer on the Model Serving team, you will primarily work with Cohere's large language models (LLMs) and other advanced NLP models that power their enterprise AI solutions. You'll be instrumental in their deployment and operation, ensuring they run efficiently and reliably.
- Does Cohere offer remote work for Site Reliability Engineers?
- Yes, Cohere is remote-friendly and has offices in several major cities. This Site Reliability Engineer position can be performed remotely, with options for co-working benefits if you prefer to work alongside others in your city.
- What technologies are most important for a Site Reliability Engineer applicant to highlight for this role?
- For this Site Reliability Engineer role, it's crucial to highlight experience with Kubernetes, large-scale production infrastructure, distributed systems, and cloud platforms like GCP, Azure, or AWS. Experience with GPU workloads and languages like Golang or C++ is also highly valued.
- How does Cohere ensure the reliability and availability of its AI models?
- Cohere ensures reliability through automated observability, resilience measures, and by defining and achieving specific Service Level Objectives (SLOs). As a Site Reliability Engineer, you will be directly involved in building and maintaining these systems, including participating in an on-call rotation to address any critical issues.
- What is the interview process like for a Site Reliability Engineer at Cohere?
- The interview process typically involves assessing your technical skills, experience with distributed systems, and problem-solving abilities. You can expect discussions on your experience with Kubernetes, cloud platforms, and potentially coding challenges in languages like Golang or C++. Behavioral questions assessing collaboration and adaptability will also be part of the process.
Similar roles
Open positions we recommend based on this role.