Senior Software Engineer, Data Ingestion Platform
RemoteHunter · United States
- Hybrid
- Full-time
- $150,000 / year
- United States
Tailored resume — keyword-matched to this role.
Hiring manager — we find who's hiring.
Intro email — drafted to reach them directly.
Job highlights
- Design next-gen data ingestion infrastructure.
- Build scalable data replication pipelines.
- Enhance Kafka Iceberg connectors.
- Modernize Change Data Capture platform.
- Develop self-service data tools.
About the role
About Our Client
The organization operates multiple brands that each address different facets of the economy to increase accessibility. These brands include platforms that facilitate commerce and financial services for sellers, enable simple money management and transactions, transform customer spending habits, empower artists as entrepreneurs, and support bitcoin management and mining. Collectively, the organization aims to build a financial system that is open and accessible to all.About the Opportunity
The Senior Software Engineer, Data Ingestion Platform will design and develop the next generation of data ingestion infrastructure within the Data Ingestion team, part of the AI, Data & Analytics division. This role is critical in ensuring the reliable and efficient flow of fresh data into the organization's Lakehouse environments to support analytics, machine learning, and AI efforts. The position influences the scalability, reliability, and cost-efficiency of the data ecosystem by modernizing ingestion platforms and consolidating data pipelines.Responsibilities
- Design, build, and operate scalable data replication and ingestion pipelines from production databases, event streams, and third-party sources.
- Develop and enhance Kafka Iceberg connectors and data loading frameworks for low-latency delivery to Snowflake and Databricks.
- Modernize the Change Data Capture platform by evaluating and implementing next-generation replication and ingestion technologies.
- Create self-service tools and observability features for teams to onboard and monitor data pipelines independently.
- Collaborate with data engineering, platform infrastructure, and product teams to define data contracts and improve service encapsulation.
- Identify opportunities to unify and streamline the data ingestion architecture, reducing complexity and redundancy.
- Design and implement privacy-compliant solutions for PII detection, masking, and data governance in line with policies and regulations such as GDPR and CCPA.
- Establish and promote best practices for pipeline reliability, cost management, schema control, and compliance.
Requirements
- 8+ years of experience in software engineering or data platform development focused on scalable data systems or distributed infrastructure.
- Proficiency in programming languages such as Java, Python, Scala, or Go with experience in data frameworks or services.
- Hands-on experience with streaming data technologies like Apache Kafka and Kafka Connect.
- Strong understanding of Change Data Capture, database replication patterns, and lake or Lakehouse architectures.
- Experience with data storage/table formats such as Apache Iceberg or Delta Lake.
- Familiarity with cloud-based data ecosystems (AWS, GCP, or Azure) and infrastructure-as-code tools.
Key skills/competency
- Data Ingestion
- Kafka
- Apache Iceberg
- Change Data Capture
- Data Pipelines
- Lakehouse Architecture
- Cloud Data Ecosystems
- Python
- Java
- Scala
Skills & topics
- Senior Software Engineer
- Data Ingestion Platform
- Data Engineering
- Software Development
- Kafka
- Apache Iceberg
- Change Data Capture
- Lakehouse
- Python
- Java
- Scala
- Cloud Data Ecosystems
- AWS
- GCP
- Azure
- Distributed Systems
- Data Pipelines
How to get hired
- Tailor your resume: Highlight your 8+ years of experience in scalable data systems, Java, Python, Scala, or Go, and Kafka expertise.
- Showcase relevant projects: Emphasize your experience with Change Data Capture, Lakehouse architectures, and cloud platforms (AWS, GCP, Azure).
- Prepare for technical questions: Be ready to discuss your understanding of data replication patterns, Iceberg/Delta Lake, and Kafka Connect.
- Demonstrate collaboration skills: Prepare examples of how you've worked with cross-functional teams to define data contracts and improve systems.
- Understand company values: Align your application and interview responses with the organization's mission of building an open financial system.
Technical preparation
Practice coding in Java, Python, Scala, or Go.,Build sample Kafka Connectors and Iceberg tables.,Set up a small-scale Lakehouse environment.,Simulate Change Data Capture scenarios.
Behavioral questions
Describe a complex data pipeline you built.,How do you ensure data pipeline reliability?,Tell me about a time you improved a system's cost-efficiency.,How do you handle data privacy compliance?
Frequently asked questions
- What are the key responsibilities for a Senior Software Engineer, Data Ingestion Platform at RemoteHunter?
- As a Senior Software Engineer, Data Ingestion Platform, you will design and build scalable data ingestion and replication pipelines, enhance Kafka Iceberg connectors, modernize the Change Data Capture platform, and develop self-service tools for data teams. You'll also focus on privacy compliance and establishing best practices for data pipelines.
- What technical skills are essential for this Senior Software Engineer role?
- Essential technical skills include 8+ years of software engineering experience, proficiency in Java, Python, Scala, or Go, hands-on experience with Apache Kafka and Kafka Connect, strong knowledge of Change Data Capture and Lakehouse architectures, and familiarity with cloud platforms like AWS, GCP, or Azure.
- How does RemoteHunter help candidates find roles like the Senior Software Engineer, Data Ingestion Platform position?
- RemoteHunter acts as a connector, helping exceptional candidates discover roles that match their goals. We guide you to complete your full application directly through the hiring company's career page or ATS, ensuring a streamlined process for roles like the Senior Software Engineer, Data Ingestion Platform.
- What is the company's mission, and how does this role contribute to it?
- The company's mission is to build a financial system that is open and accessible to all, operating various brands to achieve this. This Senior Software Engineer role is critical as it ensures the reliable flow of data needed for analytics, machine learning, and AI efforts that support these broader goals.
- What kind of data architectures are relevant for the Senior Software Engineer, Data Ingestion Platform position?
- This role heavily involves Lakehouse architectures, and experience with data storage/table formats like Apache Iceberg or Delta Lake is crucial. You'll be working with event streams, production databases, and ensuring low-latency delivery to platforms like Snowflake and Databricks.