Senior Data Engineer

Senior Data Engineer

Crisil is seeking a highly experienced Senior Data Engineer with 7 to 10 years of experience and a strong understanding of PySpark, SQL, Oracle, and PostgreSQL databases, along with proven data management expertise. This role is critical in implementing robust data architectures and driving data-driven decision-making.

Key Responsibilities

• Implement Data Architecture: Design and implement scalable, secure, and efficient data architectures on-prem and cloud platforms (Azure/GCP/AWS). Collaborate with cross-functional teams to define data requirements and develop data models.
• Data Ingestion and Integration: Develop and maintain data ingestion pipelines using PySpark, Kafka, and Flume. Integrate data from diverse sources including relational databases, NoSQL databases, APIs, and files.
• Batch and Stream Processing: Build and manage batch and stream processing pipelines using Apache Spark, Flink, and Beam. Integrate with messaging systems like Kafka, Kinesis, and Google Cloud Pub/Sub.
• SQL Proficiency: Utilize very strong SQL knowledge for query optimization, indexing, and database design.
• Delta Lake and Data Warehouse: Design and implement Delta Lake and data warehouse/mart solutions for business intelligence and analytics. Develop pipelines for data ingestion, processing, and storage.
• Distributed Databases and Data Warehousing: Implement and maintain data warehouses such as Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.
• Database Design and Development: Design, develop, and maintain scalable relational database systems (Oracle, MySQL, PostgreSQL, SQL Server). Translate business requirements into effective database solutions and optimize queries.
• Performance Tuning and Optimization: Analyze and monitor database performance, identify and resolve bottlenecks. Optimize data processing workflows and queries for improved performance and reduced latency.
• Data Management: Implement data archival and retention policies. Ensure data security and integrity through access controls, encryption, and backup strategies.
• Automation and Integration: Identify and implement automation solutions for data workflows. Integrate database solutions into software applications.
• Data Mart and Data Lake: Design and implement data marts and data lakes for BI, reporting, and analytics. Develop pipelines for data lakes like Hadoop, Amazon S3, and Azure Data Lake Storage.
• CI/CD and Automation: Develop and maintain automated testing, deployment, and monitoring scripts using Jenkins, GitLab CI/CD, or similar tools. Ensure continuous integration and delivery.
• Data Analysis and Modeling: Perform data analysis, including data modeling, mining, and visualization. Collaborate to develop models driving business insights.
• Exploration of New Tools: Explore new tools and technologies, and develop proofs-of-concept for open-source data engineering tools.
• Documentation: Document database designs, configurations, and technical specifications.

Requirements

• Bachelor's degree in Computer Science, Information Technology, or a related field. Relevant certifications are a plus.
• 7 to 10 years of experience in data engineering, with a focus on cloud-based data architectures (Azure/GCP/AWS).
• Strong proficiency in SQL and experience with relational and NoSQL databases.
• In-depth knowledge of database performance optimization techniques.
• Familiarity with data archival mechanisms and retention strategies.
• Solid understanding of database security principles.
• Experience with database administration and monitoring tools.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.
• Detail-oriented mindset.
• Ability to adapt to changing priorities and manage multiple projects.

Key skills/competency

• Data Engineering
• PySpark
• SQL
• Oracle
• PostgreSQL
• Data Management
• Cloud Platforms (Azure/GCP/AWS)
• Data Warehousing
• Data Modeling
• Performance Tuning