Data Engineer - Data Platforms Google

Introduction

A career in IBM Consulting is built on long-term client relationships and close collaboration worldwide. You will work with leading companies across industries, helping them shape their hybrid cloud and AI journeys. IBM Consulting provides support through strategic partners, robust IBM technology, and Red Hat, giving you the tools to drive meaningful change and accelerate client impact. This culture of growth and empathy values your unique skills and experiences while focusing on long-term career development.

Your Role And Responsibilities

As a Data Engineer - Data Platforms Google, you will design, build, and maintain data engineering solutions on Google's Cloud ecosystem. Your main tasks include:

• Design Data Pipelines: Develop batch and real-time pipelines for data warehouse and datalake using Google services like DataProc, DataFlow, PubSub, BigQuery, and Big Table.
• Develop Data Engineering Solutions: Utilize Google Cloud Storage, BigTable, DataProc with Spark/Hadoop, and DataFlow with Apache Beam or Python.
• Manage Data Platforms: Schedule and manage data platforms with Google Cloud Scheduler and Cloud Composer (Airflow) for efficient operations.
• Implement Data Migration: Build and execute data migration solutions ensuring seamless data transfer between systems.
• Optimize Data Layer: Design and optimize data storage and retrieval using BigQuery, Big Table, and Cloud Spanner.

Preferred Education

Master's Degree in a related field is preferred.

Required Technical And Professional Expertise

• Expertise in Google Cloud Ecosystem including DataProc, DataFlow, PubSub, BigQuery, Big Table, Cloud Spanner, CloudSQL, and AlloyDB.
• Experience in developing batch and real-time data pipelines using Apache Airflow, dbt, Spark/Python, or Spark/Scala.
• Proficiency in Google Cloud Storage, BigTable, DataProc with Spark/Hadoop, and DataFlow with Apache Beam or Python.
• Knowledge in managing data platforms using Cloud Scheduler and Cloud Composer.
• Experience in data layer design for optimal storage and retrieval.

Preferred Technical And Professional Experience

• Utilizing open-source technologies like Apache Airflow, dbt, and Spark/Python or Spark/Scala.
• Developing and implementing data migration solutions with Google services.
• Using Cloud Composer (Airflow) to manage scheduling and operations effectively.

Key skills/competency

• Google Cloud
• Data Pipelines
• Apache Beam
• BigQuery
• Data Migration
• Data Storage
• Airflow
• Spark
• Hadoop
• Cloud Composer