Data Engineer

Job Description

At Lyft, our purpose is to serve and connect. We aim to achieve this by cultivating a work environment where all team members belong and have the opportunity to thrive. As a Data Engineer on the Mapping team, you will collaborate with our world class team of engineers, product managers, and data scientists to grow and improve the quality of recommended routes and accuracy of our travel time estimations. You will architect, build, and maintain scalable data pipelines and services to support route simulation, experimentation, analytics, and machine learning models. Your work will enable integration with partner teams and allow stakeholders across Engineering, Data Science, and Product to make data-informed decisions that directly impact Lyft’s growth and profitability. Our technology stack is based on the latest technologies such as AWS, Kubernetes and Apache Airflow. You will work with incredibly passionate and talented colleagues from software engineering, machine learning and data science on projects that directly impact millions of riders and drivers. Responsibilities Owner of the core data pipelines in mapping, responsible for scaling up data processing flow to meet the rapid data growth at Lyft Develop strong subject matter expertise in the systems you manage, setting and managing SLAs for both data pipeline and datasets Continuously evolve data models and schemas to meet business and engineering requirements Develop tools that support self-service management of data pipelines (ETL), schema evolution, and perform SQL tuning to optimize data processing performance Write clean, well-tested, and maintainable code, prioritizing scalability and cost efficiency Conduct code reviews to uphold code quality standards and facilitate knowledge sharing Participate in on-call rotations to maintain high availability and reliability of data workflows Unblock, support and communicate with internal & external partners to achieve results Experiences Bachelor's degree in Computer Science, Engineering, Mathematics, Statistics, or a related field 4+ years of relevant professional experience Strong experience with Spark and with a scripting language (Python, Ruby, Bash) Exceptional analytical skills, with an in-depth understanding of data quality issues and the ability to dive into complex datasets to resolve inconsistencies Excellent communication skills, with the ability to articulate technical concepts clearly to both technical and non-technical audiences while collaborating effectively across teams Experience with disparate database, querying and streaming technologies such as S3, DynamoDB, HDFS, Hive, Presto, Pig, HBase, Parquet, Iceberg, Flink, Spark Streaming, Kafka Experience with data quality tools such as Great Expectations, dbt, Monte Carlo, Soda, Collibra St

Required Skills