Data Engineer - Data Platforms - Google

Introduction

A career in IBM Consulting is built on long-term client relationships and close global collaboration. At IBM, you will work with leading companies across industries to shape their hybrid cloud and AI journeys using innovative IBM technology and strategic partnerships.

Your Role And Responsibilities

As a Data Engineer specializing in Google data platforms, you will design, build, and maintain data engineering solutions on Google's Cloud ecosystem. Your focus areas include:

• Design Data Pipelines: Develop batch and real-time pipelines using DataProc, DataFlow, PubSub, BigQuery, and Big Table.
• Data Engineering Solutions: Utilize Cloud Storage, BigTable, DataProc with Spark/Hadoop, and DataFlow with Apache Beam or Python.
• Manage Data Platforms: Schedule and manage platforms with Cloud Scheduler and Cloud Composer (Airflow).
• Implement Data Migration: Create solutions for seamless data transfer between systems.
• Optimize Data Layer: Design and optimize storage using BigQuery, Big Table, and Cloud Spanner.

Preferred Education

Master's Degree

Required Technical And Professional Expertise

• Google Cloud Ecosystem expertise with multiple services.
• Experience in batch and real-time data pipeline development.
• Proficiency in Cloud Storage, BigTable, and DataProc with Spark/Hadoop.
• Knowledge of Cloud Scheduler and Cloud Composer for platform management.
• Understanding of optimal data layer design practices.

Preferred Technical And Professional Experience

• Experience with open-source tools like Apache Airflow, dbt, Spark/Python or Spark/Scala.
• Expertise in developing and implementing data migration solutions.
• Proficiency in scheduling and managing via Cloud Composer (Airflow).

Key skills/competency

• Google Cloud
• Data Engineering
• Data Pipelines
• BigQuery
• DataFlow
• Apache Beam
• Cloud Composer
• Data Migration
• Apache Spark
• Hadoop