Senior Data Engineer

Overview

Security is the paramount concern for Microsoft customers amidst digital threats, regulatory complexities, and intricate digital estates. Microsoft Security is dedicated to creating a safer global environment, aiming to revolutionize security by empowering every user, customer, and developer with a comprehensive security cloud that delivers end-to-end, simplified protection.

Israel serves as a pivotal center of excellence for Microsoft Security, spearheading innovation and defense against cyber threats. With over 60% of the site's focus on security, our teams develop sophisticated solutions leveraging cloud technology, big data, AI, and modern user experiences. These efforts safeguard millions worldwide and propel global cybersecurity advancements.

Our dedicated team of Data Engineers provides robust data engineering solutions crucial for generating security insights within Microsoft Security. We specialize in building scalable data pipelines, implementing advanced analytics, and integrating AI-powered anomaly detection to inform strategic decisions across the entire Security organization.

We are seeking a Senior Data Engineer who will collaborate closely with business leaders, product managers, data scientists, and analysts. In this role, you will be instrumental in developing data pipelines, datasets, data lakes, AI agents, and metrics. Your contributions will help answer critical questions regarding business performance and deliver actionable data to sustain our ongoing growth.

Responsibilities

• Design, develop, and maintain data pipelines, ETL workflows, and back-end services to support real-time decisioning, reporting, optimization, data collection, and related functions.
• Develop AI/ML workflows to facilitate queries and automate data-driven processes.
• Experiment with and recommend new technologies that can simplify or enhance the existing tech stack.
• Participate in an on-call rotation and work occasional off-hours as required.
• Mentor data engineers, provide guidance on complex design decisions, and cultivate a culture of engineering excellence and continuous learning throughout the organization.

Qualifications

• Bachelor’s degree or master’s degree in Computer Science, Math, Software Engineering, Computer Engineering, or a related field.
• You possess at least 6+ years of experience in business analytics, data science, data modeling, and/or Data Engineering.

Preferred Qualifications

• Experience architecting, building, and maintaining end-to-end, high-throughput data systems and their supporting services.
• Proficiency in designing efficient data structures and database schemas, along with experience working with distributed systems architecture.
• Highly desirable experience working with large data sets using SQL, Azure Data Lake, Spark, Databricks, etc., to derive actionable insights.
• Experience in building AI/ML-OPS tools for data platforms.
• Familiarity with developing for continuous integration and automated deployments.

Key skills/competency

• Data Engineering
• Data Pipelines
• ETL Workflows
• AI/ML Workflows
• Distributed Systems
• SQL
• Azure Data Lake
• Spark
• Databricks
• Data Modeling
• AI/ML-OPS
• Continuous Integration
• Automated Deployments
• Mentorship