Staff Software Engineer, Data Ingestion - Slack
griddable.io · Seattle, WA
- On site
- Full-time
- $180,000 / year
- Seattle, WA
Job highlights
- Build scalable data ingestion platform.
- Design streaming and Lakehouse solutions.
- Improve pipeline reliability with AI.
- Automate dataset lifecycle management.
- Lead technical initiatives and mentor engineers.
About the role
About the team
Slack is looking for a Staff Software Engineer to join the Data Ingestion Team. As part of the Data Engineering organization, we build and operate the platform that ingests data from various sources into our Lakehouse with both streaming and batch approaches. Currently, our infrastructure handles 180PB+ of storage, processes ~1T records a day, 400+ data pipelines and 1000+ Active Airflow DAGs. As Slack’s data grows (along with the number of customers, features and employees), the goal of the team is to build a highly scalable and resilient ingestion platform leveraging modern Streaming and Lakehouse technologies.
As a Staff Software Engineer, you will play a key role in driving our streaming and lakehouse strategies while also building highly scalable services and tools to help our customers acquire large volumes of high quality data at scale and making customers’ interactions with the ingested data pleasant and productive. You will have deep technical skills, be a self-starter, detail and quality oriented, and passionate about driving data driven decisions and having a huge impact at Slack!
What You Will Be Doing
- Design and develop highly scalable and resilient streaming and batch pipelines for ingesting and processing structured and unstructured datasets
- Design and implement Slack’s Lakehouse solutions and enable production use cases for customers
- Improve the reliability and performance of the services/pipelines through AI assisted and/or Agentic solutions and tools.
- Automate and handle the lifecycle of datasets (schema evolution, metadata management, change and backfill management, deprecation and migration).
- As one of the tech leads, comfortably collaborate with cross functional partners and lead technical initiatives independently end to end.
- Be a role model and a multiplier, coaching and mentoring other engineers across the team and the org.
- Write, review, or provide feedback on a technical design proposal from others.
What You Should Have
- U.S. citizenship and willing to undergo a background check for GovSlack authorization. We can not provide visa sponsorship for this role.
- 7+ years of software/data engineering experience, including experience with building applications using Streaming and Lakehouse technologies, e.g. Kafka/Kafka Connect, Flink/Spark Streaming, Iceberg/Hudi/Delta or equivalent
- Hands on experience and knowledge on building and maintaining batch data pipelines using Spark, Airflow, EMR, S3 or equivalent
- Proficient in object-oriented and/or functional programming languages: SQL, Python, Java/Scala, Go or equivalent
- Skilled at crafting and building robust distributed microservices with tools like Kubernetes, Docker, AWS ECS/EKS, Terraform, Grafana, etc.
- Familiar with AI-assisted software development and automation and have hands on experience of Claude Code/Codex or equivalent
- Excellent written and verbal communication and interpersonal skills; able to effectively collaborate with cross functional partners and explaining sophisticated technical concepts to non-technical stakeholders
For roles in San Francisco and Los Angeles: Pursuant to the San Francisco Fair Chance Ordinance and the Los Angeles Fair Chance Initiative for Hiring, Salesforce will consider for employment qualified applicants with arrest and conviction records.
Key skills/competency
- Staff Software Engineer
- Data Ingestion
- Streaming Technologies
- Lakehouse Technologies
- Data Pipelines
- Distributed Systems
- Microservices
- Kubernetes
- Python
- AWS
Skills & topics
- Staff Software Engineer
- Data Ingestion
- Streaming
- Lakehouse
- Data Pipelines
- Software Engineering
- Backend Development
- Python
- Java
- Scala
- Go
- SQL
- Kafka
- Spark
- Kubernetes
- AWS
- Airflow
- Distributed Systems
- Microservices
- Senior Engineer
How to get hired
- Tailor your resume: Highlight experience with streaming, Lakehouse technologies, and distributed systems like Kafka, Spark, and Kubernetes.
- Showcase impact: Quantify achievements in data pipeline development and microservices, focusing on scalability and reliability.
- Prepare for technical interviews: Brush up on Python, SQL, Java/Scala, and distributed systems concepts. Practice coding challenges.
- Demonstrate leadership: Be ready to discuss your experience mentoring engineers and leading technical projects end-to-end.
- Understand company culture: Research Slack's mission and values to align your responses during the interview process.
Technical preparation
Master Kafka, Spark Streaming, and Flink concepts.,Build and optimize batch pipelines with Spark, Airflow.,Implement microservices using Kubernetes and Docker.,Practice object-oriented programming in Python/Java/Scala/Go.
Behavioral questions
Describe leading a complex technical initiative.,How have you mentored junior engineers?,Explain a challenging data pipeline problem.,How do you collaborate with non-technical stakeholders?
Frequently asked questions
- What specific Streaming and Lakehouse technologies are essential for the Staff Software Engineer role at Slack?
- For the Staff Software Engineer, Data Ingestion role at Slack, proficiency in technologies like Kafka/Kafka Connect, Flink/Spark Streaming, and Lakehouse solutions such as Iceberg/Hudi/Delta is highly valued. Experience with batch processing tools like Spark, Airflow, and EMR is also crucial. Familiarity with cloud platforms like AWS is expected.
- Does Slack offer visa sponsorship for the Staff Software Engineer, Data Ingestion position?
- No, Slack does not provide visa sponsorship for this Staff Software Engineer, Data Ingestion role. Candidates must already have U.S. citizenship or be authorized to work in the U.S. without sponsorship.
- What programming languages are preferred for the Staff Software Engineer, Data Ingestion role at Slack?
- The Staff Software Engineer, Data Ingestion role at Slack requires proficiency in object-oriented and/or functional programming languages. Preferred languages include SQL, Python, Java/Scala, or Go. Python is frequently used for data engineering tasks.
- How important is experience with microservices and containerization for this Staff Software Engineer position?
- Experience with building robust distributed microservices using tools like Kubernetes, Docker, and AWS ECS/EKS is essential for this Staff Software Engineer role. Skills in infrastructure-as-code tools like Terraform are also beneficial for managing the deployment and lifecycle of these services.
- What is the expected seniority level for the Staff Software Engineer, Data Ingestion role at Slack?
- This is a Staff Software Engineer position, indicating a senior-level role. It requires 7+ years of relevant experience, including a strong track record in building scalable data systems, and the ability to lead technical initiatives and mentor other engineers.
- Are AI-assisted development tools a requirement for the Staff Software Engineer, Data Ingestion role at Slack?
- Familiarity and hands-on experience with AI-assisted software development and automation tools, such as Claude Code/Codex or equivalent, are beneficial for this Staff Software Engineer role. These tools are used to improve the reliability and performance of data pipelines.
- What kind of data volumes does the Data Ingestion team at Slack handle?
- The Data Ingestion team at Slack operates at a massive scale, handling over 180PB of storage, processing approximately 1 trillion records daily, and managing over 400 data pipelines and 1000+ active Airflow DAGs. This demonstrates the significant impact and complexity of the role.
Similar roles
Open positions we recommend based on this role.