Senior HPC DevOps Engineer

About the Role

NVIDIA is looking for an experienced Senior HPC DevOps Engineer to help build the supercomputers and HPC clusters of the future. As a key player in groundbreaking advancements in artificial intelligence and GPU computing, your expertise will drive the latest breakthroughs. You will provide insights on at-scale system design and tuning mechanisms for large-scale compute runs, working with cutting-edge Accelerated computing and Deep Learning software and hardware platforms. This role involves collaboration with scientific researchers, developers, and customers to craft improved workflows and develop new, leading differentiated solutions. You will interact with HPC, OS, GPU compute, and systems specialists to architect, develop, and bring up large-scale performance platforms.

What You’ll Be Doing

• Innovate and Implement: Design, implement, and maintain large-scale HPC/AI clusters with state-of-the-art monitoring, logging, and alerting systems.
• Infrastructure as Code (IaC): Utilize and develop tools to manage infrastructure as code, ensuring scalable and repeatable deployments.
• Streamline CI/CD Pipelines: Develop and maintain continuous integration and continuous delivery (CI/CD) pipelines to automate and streamline deployment processes.
• Automate Everything: Develop automation scripts and tools to automate deployment, configuration management, and operational monitoring, including complex networking automations.
• Troubleshoot Complex Issues: Perform comprehensive troubleshooting from bare metal to application level, ensuring system reliability and efficiency.
• Lead and Educate: Serve as a technical resource, developing and sharing best practices with internal teams.
• Drive Innovation: Support R&D activities and engage in proof of concepts (POCs) and proof of values (POVs) for future improvements.

What We Need To See

• B.Sc. in Computer Science, Engineering, or a related field with 5+ years of experience.
• Deep knowledge of HPC and AI solution technologies, including CPUs, GPUs, high-speed interconnects, and supporting software.
• Advanced proficiency in programming and scripting languages, with a solid understanding of object-oriented programming principles.
• Familiarity with Jenkins, Ansible, Puppet/Chef.
• Excellent knowledge of Windows and Linux (Redhat/CentOS and Ubuntu), networking and OS-level security.
• Deep understanding of networking protocols such as InfiniBand and Ethernet.
• Experience with job scheduling workloads and orchestration tools such as Slurm and Kubernetes.
• Background with multiple storage solutions like Lustre, GPFS, ZFS, and XFS.
• Expertise with virtual systems (VMware, Hyper-V, KVM, Citrix).
• Familiarity with cloud platforms (AWS, Azure, Google Cloud).

Ways To Stand Out From The Crowd

• Proven networking experience or strong knowledge through professional networking training.
• Architectural Insight: Knowledge of CPU and/or GPU architecture.
• Container Expertise: Understanding of Kubernetes and container-related microservice technologies.
• GPU Focus: Experience with GPU-focused hardware/software (DGX, CUDA).
• RDMA Fabrics: Background with RDMA (InfiniBand or RoCE) fabrics.

Key skills/competency

• HPC DevOps
• AI Clusters
• Infrastructure as Code (IaC)
• CI/CD Pipelines
• Automation Scripting
• Networking Protocols (InfiniBand, Ethernet)
• Linux Administration
• Kubernetes & Slurm
• GPU Computing
• Troubleshooting