Senior Observability & Telemetry Engineer - Radian Arc
Submer · EMEA
- Hybrid
- Full-time
- $150,000 / year
- EMEA
Job highlights
- Design and build observability platform for GPU cloud.
- Implement telemetry pipelines for metrics, logs, traces.
- Ensure visibility across infrastructure and workloads.
- Collaborate with engineering and operations teams.
- Drive improvements in platform reliability and performance.
About the role
About Radian Arc
Radian Arc, now part of InferX, Submer's AI cloud and GPU infrastructure platform, provides an infrastructure-as-a-service (IaaS) platform for running cloud gaming, artificial intelligence and machine learning applications inside telecommunication carrier networks. Our teams across the USA, Australia, Central Europe, Malaysia, Singapore and Japan offer telecom operators a GPU-based edge computing platform without the need for capital expenditure, facilitating low latency and improved economics for value-added services and the monetization of 5G investments.What Impact You Will Have
Mission: Design and build the observability platform that powers visibility, reliability, and performance insights for large-scale GPU cloud infrastructure as well as smaller edge deployments. This role is responsible for designing and implementing key parts of the observability architecture across the platform, enabling engineering, operations, and customers to understand system behavior in real time across distributed AI workloads, GPU clusters, networking fabrics, storage systems, and edge inference environments. You will design and operate low-latency, high-scale telemetry pipelines that collect, process, and analyze metrics, logs, and traces from infrastructure running across core datacenter clusters and smaller edge deployments. The platform you build will support internal operations, automated reliability mechanisms, and customer-facing observability experiences. As a senior engineer, you will lead delivery of major observability initiatives, contribute to the evolution of telemetry standards and SLO implementation, and work with other teams to ensure observability is effectively integrated into the platform architecture from infrastructure to application layers. You will collaborate closely with infrastructure, networking, storage, and platform engineering teams to provide clear visibility into performance bottlenecks, infrastructure degradation, and distributed workload behavior across both hyperscale GPU environments and smaller edge installations. This role contributes directly to improving platform reliability by analyzing production telemetry, identifying systemic issues, and driving improvements in performance, efficiency, and operational stability across the stack.What You’ll Do
Observability Platform Architecture
- Design and implement scalable telemetry pipelines for metrics, logs, and traces across distributed GPU infrastructure.
- Architect observability systems capable of ingesting high-cardinality telemetry from thousands of nodes and services.
- Build and operate telemetry storage systems optimized for large-scale time-series and event data.
- Contribute to observability standards across services, including metrics, tracing instrumentation, logging, and SLO implementation.
Infrastructure and Platform Observability
- Build visibility across compute, storage, and networking layers of the platform.
- Instrument GPU clusters, inference workloads, and distributed training environments.
- Detect infrastructure degradation such as: GPU throttling, Network congestion, Storage latency, Hardware degradation.
- Implement telemetry pipelines for GPU, CPU, network, and storage performance metrics.
Customer-Facing Observability
- Build dashboards and monitoring tools that expose system health and performance to both internal teams and customers.
- Provide insights into workload performance including: GPU utilization, Storage throughput, Network latency, Distributed inference performance.
- Develop performance analysis tools that help customers understand system bottlenecks.
Network and Infrastructure Telemetry
- Develop and maintain network observability platforms.
- Build telemetry collectors and exporters using Python or Go.
- Ingest telemetry from infrastructure components including: NVIDIA Cumulus Linux, VyOS routers, Citrix NetScaler / WAF.
- Design telemetry ingestion pipelines using protocols such as: gNMI, SNMP, Streaming telemetry.
Reliability Engineering
- Design advanced alerting and anomaly detection systems.
- Contribute to platform SLOs, SLIs, and reliability metrics.
- Build automated detection of infrastructure anomalies.
- Integrate observability signals with operational workflows and incident management systems.
- Participate in on-call rotations supporting platform observability and telemetry infrastructure.
Cross-Team Collaboration
- Partner with platform, networking, storage, and compute teams to instrument services.
- Work closely with operations teams to improve monitoring and incident response.
- Provide guidance and mentorship to engineers on observability best practices.
- Promote good observability practices across teams and help engineers adopt effective instrumentation and monitoring patterns.
Technical Stack: Observability and telemetry technologies used across the platform include:
Observability Framework
- Prometheus.
- OpenTelemetry.
- Grafana.
- Distributed logging systems.
- High-scale telemetry databases, such as ClickHouse or similar.
Hardware and Infrastructure Telemetry
- Redfish / BMC telemetry.
- IPMI.
- Linux system metrics.
- Hardware health monitoring and node lifecycle telemetry.
NVIDIA GPU Telemetry
- NVIDIA DCGM.
- DCGM Exporter.
- NVML.
- NVIDIA GPU Operator telemetry stack.
- NVSwitch / NVLink telemetry.
AI Workload Telemetry
- Distributed training telemetry.
- Inference latency and throughput metrics.
- NCCL communication health.
- GPU synchronization latency.
- KV-cache access latency for inference workloads.
- Dataset loading and storage I/O performance.
Networking Telemetry
- NVIDIA NetQ.
- gNMI streaming telemetry.
- SNMP.
- Network flow telemetry.
- RDMA / RoCE performance monitoring.
What You’ll Need
Required Experience
- Proven experience operating large distributed infrastructure platforms.
- Strong background in observability systems and telemetry pipelines.
- Experience building metrics, logging, tracing, alerting, and dashboards at production scale.
- Strong programming skills in Go, Python, or Rust.
- Experience with large-scale time-series data platforms.
- Experience with large-scale GPU cloud platforms, HPC environments, or AI infrastructure.
- Experience monitoring AI workloads such as training or inference clusters.
Infrastructure Knowledge
- Deep understanding of distributed systems observability.
- Familiarity with cloud-native infrastructure such as Kubernetes, automation, and CI/CD.
- Experience operating observability systems for high-performance or large-scale environments.
Networking and Infrastructure Telemetry
- Experience monitoring complex networking environments.
- Familiarity with telemetry protocols such as gNMI, SNMP, and streaming telemetry.
- Experience integrating network and system telemetry into centralized monitoring platforms.
Analytical Skills
- Strong data analysis capabilities.
- Ability to interpret complex telemetry signals and translate them into actionable insights.
- Ability to diagnose performance issues across distributed systems.
What We Offer
Attractive compensation package reflecting your expertise and experience. A great work environment characterised by friendliness, international diversity, flexibility, and a hybrid-friendly approach. You'll be part of a fast-growing scale-up with a mission to make a positive impact, offering an exciting career evolution. Our job titles may span more than one job level. The actual base pay is dependent on a number of factors, such as transferable skills, work experience, business needs and market demands.Our Inclusive Responsibility
Radian Arc is committed to creating a diverse and inclusive environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, veteran status, or any other protected category under applicable law. From concept to deployment, we create liquid-cooled datacenters that scale smarter, run cleaner and cost less.Skills & topics
- Observability
- Telemetry
- SRE
- Site Reliability Engineering
- GPU
- AI
- Machine Learning
- Cloud Infrastructure
- Distributed Systems
- Prometheus
- OpenTelemetry
- Grafana
- Go
- Python
- Rust
- Cloud Gaming
- Edge Computing
- Kubernetes
- Performance Monitoring
- Networking
How to get hired
- Tailor your resume: Highlight experience with observability, telemetry, Go/Python/Rust, and large-scale infrastructure.
- Showcase your impact: Quantify achievements in building and operating monitoring systems for production environments.
- Prepare for technical interviews: Be ready to discuss distributed systems, telemetry protocols, and performance analysis.
- Demonstrate collaboration: Emphasize your ability to work with diverse engineering teams.
- Research Radian Arc: Understand their mission and the role of AI/ML in telecommunication networks.
Technical preparation
Master Go, Python, or Rust for telemetry.,Understand distributed systems observability deeply.,Practice with Prometheus, OpenTelemetry, Grafana.,Familiarize with GPU, AI workload monitoring.
Behavioral questions
Describe a complex system you monitored.,How do you handle production incidents?,How do you mentor other engineers?,How do you collaborate with cross-functional teams?
Frequently asked questions
- What is the work arrangement for the Senior Observability & Telemetry Engineer role at Radian Arc?
- The Senior Observability & Telemetry Engineer role at Radian Arc is a remote position within the EMEA region. While the company mentions a 'hybrid-friendly approach,' this specific role is advertised as remote.
- What are the key responsibilities of a Senior Observability & Telemetry Engineer at Radian Arc?
- Key responsibilities include designing and implementing the observability platform for large-scale GPU cloud infrastructure, developing low-latency telemetry pipelines for metrics, logs, and traces, and collaborating with engineering teams to ensure system visibility and reliability.
- What programming languages are essential for the Senior Observability & Telemetry Engineer position at Radian Arc?
- Strong programming skills in Go, Python, or Rust are required for this role. Proficiency in at least one of these languages is crucial for building telemetry collectors, exporters, and other observability tools.
- Does Radian Arc require experience with specific hardware like NVIDIA GPUs for this role?
- Yes, experience with large-scale GPU cloud platforms, HPC environments, or AI infrastructure, including monitoring AI workloads and specific NVIDIA GPU telemetry (DCGM, NVML), is a significant requirement for the Senior Observability & Telemetry Engineer position.
- What kind of data analysis skills are needed for this role at Radian Arc?
- Strong data analysis capabilities are essential. You'll need to interpret complex telemetry signals, diagnose performance issues across distributed systems, and translate data into actionable insights for improving platform reliability and performance.
- How does Radian Arc support career evolution for its employees?
- Radian Arc emphasizes career evolution by being a fast-growing scale-up with a mission-driven approach. They offer opportunities for professional development and growth within a dynamic environment, as highlighted in their 'What We Offer' section.
- What are the main observability technologies used at Radian Arc?
- Radian Arc utilizes a range of observability technologies, including Prometheus, OpenTelemetry, Grafana, distributed logging systems, and high-scale telemetry databases like ClickHouse. They also leverage specific tools for hardware, NVIDIA GPU, AI workload, and networking telemetry.