Azure Databricks Platform Administrator

About Capgemini and the Azure Databricks Platform Administrator Role

Choosing Capgemini means choosing a company where you will be empowered to shape your career, supported by a collaborative global community, and inspired to reimagine what’s possible. Join us in helping the world’s leading organizations unlock the value of technology and build a more sustainable, inclusive world.

Capgemini is urgently seeking a Platform Engineer for enterprise-level metadata management and data access initiatives. The ideal candidate will possess deep expertise in platform administration of Databricks and Unity Catalog, driving strategic alignment across technical and business teams. This role offers the opportunity to shape scalable, secure data ecosystems and is open to relocation.

Your Role and Responsibilities

• Implement data provisioning patterns based on business requirements, adhering to predefined processes, policies, standards, and metadata management rules.
• Create and manage distributed workspaces in Databricks, setting up workspace policies, provisioning Databricks clusters, and managing data infrastructure sizing and capacity.
• Develop Python notebooks, implement data masking processes, create UDFs (SQL/Python), and troubleshoot data pipelines.
• Ensure data security and compliance with regulations using Databricks and Privacera's features.
• Navigate multi-step enterprise approval processes across architecture, security, and governance teams.
• Design and implement data architecture leveraging technologies such as Databricks, Unity Catalog, and Privacera.
• Develop, optimize, and manage data pipelines for ETL processes using Databricks, with a focus on data integrity and quality.
• Design and maintain data models and schemas, incorporating Unity Catalog and Collibra data governance practices.
• Establish a robust data governance strategy, defining standards, metadata management, lineage, and quality practices.
• Operationalize Machine Learning models in Batch and Real Time Data Pipelines, leveraging relevant governance setups.
• Collaborate with cross-functional teams including data scientists, engineers, and analysts to translate business requirements into scalable solutions.
• Foster collaboration and clarity in complex, ambiguous environments.

Your Skills and Experience

• Extensive hands-on platform engineering experience with Databricks and Unity Catalog.
• Proven success in implementing distributed Databricks workspaces, SQL, and Python scripting, and automation.
• Experience with Azure infrastructure provisioning.
• Familiarity with data governance frameworks and compliance standards.
• Familiarity with Privacera data security and access control management.
• Familiarity with data requirements of common ML/AI use cases.
• Awareness of data governance frameworks, enterprise data compliance requirements, metadata modeling, data architecture, and enterprise-scale data discovery solutions.

Key Skills/Competency

• Databricks Administration
• Unity Catalog
• Azure Infrastructure
• Data Governance
• SQL & Python Scripting
• Data Pipelines (ETL)
• Metadata Management
• Privacera Security
• MLOps (Machine Learning Operations)
• Cloud Platform Engineering