Senior Engineer - Databricks

About The Role

As a Senior Engineer specializing in Databricks, you will serve as a subject matter expert and innovative mentor, translating data strategy into effective IT solutions while fostering data‑driven thinking across Shell’s broader stakeholder community. You will be part of Shell’s global Trading and Supply organization, the world’s largest energy trader, supporting its integrated network for crude, products, gas, LNG, power, and environmental products across major trading hubs. Within the Trading and Supply CC, you’ll work in long‑standing, business‑focused teams that deliver customer‑centric digital solutions, enhance risk and control management, and enable progress in Shell’s energy transition. Supporting the Crude and Products businesses, you will help ensure safe, efficient delivery of services in a complex, highly regulated environment, simplifying and modernizing digital capabilities while contributing to an inclusive, collaborative team culture.

What You’ll Be Doing

• Serve as a technology champion with strong expertise in relational and data‑warehouse systems, advanced SQL query authoring, and familiarity with diverse database technologies; experience with NoSQL and end‑to‑end pipeline development is a strong advantage.
• Build and optimize complex queries, and efficiently manipulate, process, and extract value from large, diverse datasets.
• Work with key data engineering technologies such as ADLS, ADF, Azure Databricks, Azure SQL, Synapse, or SAP HANA, applying your big‑data experience to deliver scalable solutions.
• Leverage experience with Delta Live Tables (DLT) and structured streaming in Databricks (preferred).
• Apply deep expertise in scripting and programming using Python, SQL, Spark SQL, or PySpark.
• Demonstrate strong understanding of Data Foundation practices, including data modeling, data quality management, governance, maturity assessments, and data strategy.
• Build trusted relationships and communicate effectively with developers, architects, and business stakeholders to influence outcomes and drive alignment.
• Lead the rollout and adoption of Data Foundation initiatives by advising on technology, applying best‑practice tools, and supporting critical business programs.
• Coordinate change management, incident management, and problem management activities to maintain stability and operational excellence.
• Ensure end‑to‑end requirements traceability from data and testing to scope changes, training, and transition to maintain clear alignment throughout delivery.
• Drive efficiency across pilots and future projects to reduce costs, speed up implementation, and maximize business value.
• Take ownership of initiatives, encourage innovation, and independently lead new ideas or solutions to successful execution.

What You Bring

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• 9-12 years of experience in Data Engineering with deep expertise in Databricks, including Spark architecture, Spark execution engine, Delta Live Tables (DLT), PySpark, SQL, Mosaic AI, and performance/cost optimization.
• Strong core Python programming skills.
• Hands-on experience with Azure services such as Azure Functions, ADF Data Flows, Logic Apps, and FAST/REST APIs.
• Experience building and enhancing CI/CD pipelines for build and release automation.
• Strong analytical, logical reasoning, and problem‑solving capabilities.
• A curious mindset with a willingness to learn, ask questions, and actively engage in discussions.
• Excellent communication skills, including business communication and storytelling abilities.
• Strong stakeholder management and collaboration skills.
• Passion for technology innovation, with a proactive approach to taking ownership and driving initiatives to completion.
• Exposure to scripting languages such as YAML, PowerShell, or Terraform (a plus).
• Experience with big data technologies such as Kafka, Hadoop, and Spark, with familiarity in stream‑processing systems (nice to have).
• Experience using data pipeline and workflow orchestration tools like Azkaban, Luigi, or Airflow (nice to have).
• Additional Python skills with libraries such as NumPy, Pandas, or Django (nice to have).
• Familiarity with visualization tools such as Power BI or Qlik Sense (nice to have).
• Knowledge of data science concepts (AI/ML, NLP, GenAI) is advantageous.
• Azure, Databricks Professional, or Data Science certifications are a plus.

What We Offer

You bring your skills and experience to Shell and in return you work with talented, committed people on one of the most important challenges facing our planet. You’ll have the opportunity to develop the skills you need to grow in an environment where we value honesty, integrity, and respect for one another. You’ll be able to balance your priorities as you become the best version of yourself.

• Progress as a person as we work on the energy transition together.
• Continuously grow the transferable skills you need to get ahead.
• Work at the forefront of technology, trends, and practices.
• Collaborate with experienced colleagues with unique expertise.
• Achieve your balance in a value-led culture that encourages you to be the best version of yourself.
• Benefit from flexible working hours, and the possibility of remote/mobile working.
• Perform at your best with a competitive starting salary and annual performance related salary increase – our pay and benefits packages are among the best in the world.
• Take advantage of paid parental leave, including for non-birthing parents.
• Join an organization to become one of the most diverse and inclusive in the world. We strongly encourage applicants of all genders, ages, ethnicities, cultures, abilities, sexual orientation, and life experiences to apply.
• Grow as you progress through diverse career opportunities in national and international teams.
• Gain access to a wide range of training and development programs.

We'd like you to know that Shell has a bold goal: to become one of the world’s most diverse and inclusive companies. You can get to know more about how we're working towards that goal, click here.

IMPORTANT NOTE: After you submit your application, you will receive an email providing a link to an online assessment that is a mandatory part of the process. The email will come from HireVue - please check your spam/junk mailbox if you do not see the email. Once completed, your application will be reviewed to determine next steps. Thank you for your interest in Shell.

Key skills/competency

• Databricks
• Data Engineering
• Python
• SQL
• Azure
• Spark
• Data Modeling
• CI/CD
• Stakeholder Management
• Problem-solving