Senior Data Engineer
Crisil · Kurla, Maharashtra, India
This listing has closed — view similar roles below.
- On site
- Full-time
- ₹2,500,000 / year
- Kurla, Maharashtra, India
Job highlights
- Implement robust data architectures on cloud platforms.
- Develop and maintain data ingestion and processing pipelines.
- Utilize strong SQL for query optimization and design.
- Design and implement Delta Lake and data warehouses.
- Ensure data integrity, security, and performance.
About the role
Senior Data Engineer
Crisil is seeking a highly experienced Senior Data Engineer with 7 to 10 years of experience and a strong understanding of PySpark, SQL, Oracle, and PostgreSQL databases, along with proven data management expertise. This role is critical in implementing robust data architectures and driving data-driven decision-making.
Key Responsibilities
- Implement Data Architecture: Design and implement scalable, secure, and efficient data architectures on-prem and cloud platforms (Azure/GCP/AWS). Collaborate with cross-functional teams to define data requirements and develop data models.
- Data Ingestion and Integration: Develop and maintain data ingestion pipelines using PySpark, Kafka, and Flume. Integrate data from diverse sources including relational databases, NoSQL databases, APIs, and files.
- Batch and Stream Processing: Build and manage batch and stream processing pipelines using Apache Spark, Flink, and Beam. Integrate with messaging systems like Kafka, Kinesis, and Google Cloud Pub/Sub.
- SQL Proficiency: Utilize very strong SQL knowledge for query optimization, indexing, and database design.
- Delta Lake and Data Warehouse: Design and implement Delta Lake and data warehouse/mart solutions for business intelligence and analytics. Develop pipelines for data ingestion, processing, and storage.
- Distributed Databases and Data Warehousing: Implement and maintain data warehouses such as Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.
- Database Design and Development: Design, develop, and maintain scalable relational database systems (Oracle, MySQL, PostgreSQL, SQL Server). Translate business requirements into effective database solutions and optimize queries.
- Performance Tuning and Optimization: Analyze and monitor database performance, identify and resolve bottlenecks. Optimize data processing workflows and queries for improved performance and reduced latency.
- Data Management: Implement data archival and retention policies. Ensure data security and integrity through access controls, encryption, and backup strategies.
- Automation and Integration: Identify and implement automation solutions for data workflows. Integrate database solutions into software applications.
- Data Mart and Data Lake: Design and implement data marts and data lakes for BI, reporting, and analytics. Develop pipelines for data lakes like Hadoop, Amazon S3, and Azure Data Lake Storage.
- CI/CD and Automation: Develop and maintain automated testing, deployment, and monitoring scripts using Jenkins, GitLab CI/CD, or similar tools. Ensure continuous integration and delivery.
- Data Analysis and Modeling: Perform data analysis, including data modeling, mining, and visualization. Collaborate to develop models driving business insights.
- Exploration of New Tools: Explore new tools and technologies, and develop proofs-of-concept for open-source data engineering tools.
- Documentation: Document database designs, configurations, and technical specifications.
Requirements
- Bachelor's degree in Computer Science, Information Technology, or a related field. Relevant certifications are a plus.
- 7 to 10 years of experience in data engineering, with a focus on cloud-based data architectures (Azure/GCP/AWS).
- Strong proficiency in SQL and experience with relational and NoSQL databases.
- In-depth knowledge of database performance optimization techniques.
- Familiarity with data archival mechanisms and retention strategies.
- Solid understanding of database security principles.
- Experience with database administration and monitoring tools.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration skills.
- Detail-oriented mindset.
- Ability to adapt to changing priorities and manage multiple projects.
Key skills/competency
- Data Engineering
- PySpark
- SQL
- Oracle
- PostgreSQL
- Data Management
- Cloud Platforms (Azure/GCP/AWS)
- Data Warehousing
- Data Modeling
- Performance Tuning
Skills & topics
- Data Engineer
- Senior Data Engineer
- Data Architecture
- Data Ingestion
- Data Integration
- PySpark
- SQL
- Oracle
- PostgreSQL
- Cloud Data Platforms
- Azure
- GCP
- AWS
- Data Warehousing
- Delta Lake
- Data Management
- Performance Tuning
- CI/CD
- Data Analysis
- Data Modeling
How to get hired
- Tailor your resume: Highlight your 7-10 years of experience in data engineering, focusing on PySpark, SQL, Oracle, PostgreSQL, and cloud platforms like Azure/GCP/AWS.
- Showcase cloud architecture expertise: Emphasize your experience in implementing scalable and secure data architectures on Azure, GCP, or AWS.
- Demonstrate SQL and database skills: Provide specific examples of your strong SQL knowledge, including query optimization and experience with relational and NoSQL databases.
- Quantify achievements: Use data and metrics to illustrate the impact of your work in areas like performance tuning, data management, and CI/CD implementation.
Technical preparation
Master PySpark and SQL for complex data tasks.,Practice designing scalable cloud data architectures.,Prepare to demonstrate database optimization skills.,Familiarize with CI/CD tools for data pipelines.
Behavioral questions
Describe a complex data problem you solved.,How do you ensure data integrity and security?,Discuss your experience collaborating with teams.,How do you stay updated with new technologies?
Frequently asked questions
- What are the primary cloud platforms mentioned for the Senior Data Engineer role at Crisil?
- The Senior Data Engineer role at Crisil emphasizes experience with cloud-based data architectures on Azure, GCP, and AWS. Candidates are expected to have implemented scalable, secure, and efficient data solutions on these platforms.
- What specific database technologies are crucial for this Senior Data Engineer position?
- This role requires strong proficiency in SQL and experience with various database management systems. Specifically, Oracle and PostgreSQL are mentioned, along with a general requirement for both relational and NoSQL database experience.
- What level of experience is required for the Senior Data Engineer at Crisil?
- Crisil is looking for a Senior Data Engineer with 7 to 10 years of experience in data engineering. The focus is on candidates who have led projects and possess a deep understanding of cloud-based data architectures.
- Does Crisil require specific certifications for the Senior Data Engineer role?
- While a Bachelor's degree in Computer Science, Information Technology, or a related field is required, relevant certifications are considered a plus for the Senior Data Engineer position. They can help demonstrate specialized knowledge.
- What are the expectations for data processing methods in this Senior Data Engineer role?
- The Senior Data Engineer will be responsible for developing and maintaining both batch and stream processing pipelines. Experience with tools like Apache Spark, Apache Flink, and Apache Beam, along with integration with messaging systems like Kafka, is expected.