Senior Data Engineer

About Shift Technology

Shift is the leading AI platform for insurance. Shift combines generative, agentic, and predictive AI to transform underwriting, claims, and fraud and risk - driving operational efficiency, exceptional customer experiences and measurable business impact. Trusted by the world's leading insurers, Shift delivers AI when and where it matters most, at scale and with proven results. Our culture is built on innovation, trust, and a drive to transform the insurance industry through our SaaS platform. We come from more than 50 different countries and cultures and together we are creating the future of insurance. Our Engineering team lies at the core of the value we offer to our customers. We solve complex problems not only within our squads but also by working collaboratively with other teams across the organization. We’re reshaping the way data drives decision-making by building powerful, scalable software and data platforms. Whether you're passionate about distributed systems, building APIs, or enabling data-driven insights, you'll find exciting opportunities here. Our tech stack includes .NET Core (C#), Python, MS SQL Server, Spark, Azure Data Factory , Databricks, GoodData , terraform and more. We are also progressing towards distributed data systems on cloud platforms, with exciting challenges in both batch and real-time processing.

What You'll Do

• Drive end-to-end data architecture design for scalable and resilient systems.
• Design, implement, and maintain robust ETL/ELT pipelines.
• Own and optimize Apache Spark-based pipelines (debugging, hardening, tuning, performance optimization).
• Translate business requirements into scalable data models and warehouse schemas.
• Improve pipeline performance, reliability, scalability, and cost efficiency.
• Lead or contribute to POCs and new technology adoption initiatives.
• Ensure best practices in data governance, quality, observability, and monitoring.
• Work closely with cross-functional teams (Analytics, ML, Product, Platform).
• Mentor and support junior engineers, providing technical guidance and code reviews.
• Take ownership of infrastructure provisioning and unblock yourself when needed.
• Actively document and diagram the technical design and architecture.

What You Bring

• 8+ years experience, Tech Lead experience is a plus.
• Drive software architecture, understand the data model/ETL mapping, contribute to new tech adoption (POCs), and mentor peers.
• Apache Spark a must, with experience debugging, hardening and optimizing pipelines.
• 3+ years of SQL database, CosmosDB.
• 3+ years experience in custom ETL design, implementation and maintenance.
• 3+ years experience with Data Modeling.
• Strong understanding of: Distributed data systems, Data warehousing concepts, Batch and event-driven processing, Performance and scalability troubleshooting.
• Hands-on experience with: Databricks, DBT, Infrastructure as Code (Terraform, ARM, Bicep, etc.), Azure data services (Event Hub, Service Bus, ADLS, Synapse, etc.).
• Experience provisioning infrastructure and setting up data pipelines independently.
• Bonus Experience with stores: DBT, Analytical tools, feature store, vector database.
• Nice to have: OLAP db, GraphDB, / SQL mesh (Niche).

Hiring Process

• Talent Acquisition interview - 30 minutes
• Hiring Manager interview - 45 minutes
• Technical Screening with an Architect - 45 to 60 minutes
• Second Technical Screening with a Tech Lead or Data Engineer - 45 to 60 minutes
• Final Round with Hiring Manager - 60 minutes

Benefits

Shift Technology offers a competitive total rewards and benefits package to support permanent, full-time employees. Highlights include:

• Flexible remote and hybrid working options
• Competitive Salary and a variable component tied to personal and company performance
• Multiple Learning and Development opportunities, including Focus Fridays
• Generous PTO and paid holidays
• Mental health benefits
• 2 MAD Days per year (Make A Difference Days for paid volunteering)

Additional benefits may be offered by country. Intern and Apprentice positions may receive some of these benefits. Shift Technology strives to be a diverse and inclusive workforce.

Key skills/competency

• Data Engineering
• Python
• Terraform
• Apache Spark
• ETL/ELT
• Data Modeling
• Cloud Platforms (Azure)
• SQL
• Databricks
• Infrastructure as Code