Data Engineer
Tek Leaders Inc
Job Overview
Who's the hiring manager?
Sign up to PitchMeAI to discover the hiring manager's details for this job. We will also write them an intro email for you.
Job Description
About the Opportunity
Our client, a leader in the Analytics and AI space, is seeking two Data Engineers to join a specialized team focused on product-agnostic data preprocessing. This team is responsible for transforming data upstream before it feeds into various core products. The project primarily involves a strategic migration from a legacy platform (Informatica sunset) to a modern Spark-based environment. This is a high-impact role for a self-motivated professional who can deliver results in a fast-paced, evolving technical landscape.
Key Responsibilities & Deliverables
- Legacy Code Migration: Reviewing and interpreting legacy code (including Scala) to refactor and rewrite into Python/PySpark for the new platform.
- Pipeline Development: Building and maintaining a new data migration module and data pipelines for both historical data resets and ongoing batch ingestion.
- Data Transformation: Developing product-agnostic data layers to ensure clean data flow across multiple internal products.
- Cloud Data Management: Utilizing AWS Glue for Spark jobs, Lambda for serverless functions, and managing data files within S3 buckets.
- Quality Assurance: Ensuring data integrity through rigorous validation and end-to-end testing during the baseline reset for migrated clients.
Required Skills & Experience
We are looking for technical experts who can hit the ground running with the following skills:
- PySpark Mastery: Deep expertise in Python and PySpark (Dataframes) for data transformation and ETL.
- Code Refactoring: The ability to read and understand Scala for the purpose of rewriting it into Python (actual coding will be in Python/PySpark).
- Core AWS Stack: Strong hands-on experience with AWS Glue, S3, and Lambda. Familiarity with other AWS data tools is a significant plus.
- ETL Infrastructure: Proven experience building multi-layer pipelines (Raw to Cleansed) and handling complex schema mapping.
- Analytical Rigor: Strong SQL skills for data validation and reconciliation.
Key skills/competency
- PySpark
- Python
- AWS Glue
- S3
- Lambda
- Scala (reading)
- SQL
- ETL
- Data Migration
- Data Transformation
How to Get Hired at Tek Leaders Inc
- Research Tek Leaders Inc's culture: Study their mission, values, recent news, and employee testimonials on LinkedIn and Glassdoor.
- Tailor your resume: Customize your resume and cover letter to highlight PySpark, AWS Glue, and data migration expertise relevant to this Data Engineer role.
- Showcase technical projects: Prepare to discuss past data engineering projects, especially those involving Scala to Python migration and multi-layer ETL pipelines.
- Master behavioral questions: Practice answering common behavioral questions, focusing on problem-solving, collaboration, and adapting to fast-paced technical changes.
- Network effectively: Connect with current and former Tek Leaders Inc employees on LinkedIn to gain insights into the company and team dynamics.
Frequently Asked Questions
Find answers to common questions about this job opportunity
Explore similar opportunities that match your background