Data Engineer

About Lyft

At Lyft, our purpose is to serve and connect. We aim to achieve this by cultivating a work environment where all team members belong and have the opportunity to thrive.

Our transport network serves the needs of millions of people every day who want to get from one place to another using Lyft cars, bikes and scooters, with public transportation, or on foot in the most efficient way. To serve these needs, we need to suggest the fastest, most affordable and safest routes. We achieve this by processing millions of rides, taking into account the latest traffic information and analyzing the preferences of drivers.

The Role: Data Engineer

To strengthen our efforts, we are hiring a Data Engineer to help us make data-driven decisions. Data Engineering is at the heart of Lyft’s products and decision-making. As a Data Engineer at Lyft, you will be tasked with developing robust data infrastructure—encompassing data transport, collection, and storage—and providing services that enable our leadership to make informed, risk-reducing decisions.

You will help architect, build, and launch scalable data pipelines to support Lyft’s growing data processing and analytics needs. Your efforts will allow access to business and user behavior insights, using huge amounts of Lyft data to fuel several teams such as Analytics, Data Science, Engineering, and many others.

Our technology stack is based on the latest technologies such as AWS, Kubernetes and Apache Airflow. You will work with incredibly passionate and talented colleagues from software engineering, machine learning and data science on projects that delight millions of passengers and drivers.

Responsibilities

• Owner of the core data pipelines in mapping, responsible for scaling up data processing flow to meet the rapid data growth at Lyft.
• Evolve data model and data schema based on business and engineering needs.
• Design, develop and deploy tooling and systems that continually improve the reliability, scalability and efficiency of our platform.
• Implement systems tracking data quality and consistency.
• Develop tools supporting self-service data pipeline management (ETL).
• SQL and MapReduce job tuning to improve data processing performance.
• Write well-crafted, well-tested, readable, maintainable code.
• Participate in code reviews to ensure code quality and distribute knowledge.
• Participate in on-call rotations to ensure high availability and reliability of workflows and data.
• Unblock, support and communicate with internal & external partners to achieve results.

Experience

• Bachelor's degree in Computer Science, Engineering, Mathematics, Statistics, or a related field.
• 2+ years of relevant professional experience.
• Strong experience with Spark.
• Experience with Hadoop (or similar) Ecosystem, S3, DynamoDB, MapReduce, Yarn, HDFS, Hive, Spark, Presto, Pig, HBase, Parquet.
• Experience defining API schemas and developing backend services in a microservices environment.
• Strong skills in a scripting language (Python, Ruby, Bash).
• Good understanding of SQL Engine and able to conduct advanced performance tuning.
• Proficient in at least one of the SQL languages (MySQL, PostgreSQL, SqlServer, Oracle).
• Experience with workflow management tools (Airflow, Oozie, Azkaban, UC4).
• Comfortable working directly with data and business partners to bridge Lyft’s business goals with data engineering.

Key skills/competency

• Data Engineering
• Scalable Data Pipelines
• Spark
• Hadoop Ecosystem
• SQL Tuning
• AWS
• Kubernetes
• Apache Airflow
• Python
• Microservices