Solution Lead – Big Data and Data Engineering
Cathay Pacific · Hong Kong, Hong Kong SAR
This listing has closed — view similar roles below.
- On site
- Full-time
- HK$600,000 / year
- Hong Kong, Hong Kong SAR
Job highlights
- Lead big data and ML infrastructure design and implementation.
- Develop and support large-scale data pipelines on AWS.
- Collaborate with diverse technical teams and stakeholders.
- Implement CI/CD and containerization for ML applications.
- Drive innovation in agile, data-driven environments.
About the role
Role Introduction
Join Cathay as a Solution Lead in our Business Intelligence Competency Centre (BICC) Solution Centre. You will provide subject-matter technical competence in designing, implementing, maintaining, and administrating big data and machine learning infrastructure, defining solutions to bridge gaps and ensuring the project scope is maintained, and solutions implemented in line with design.
Thrive in a dynamic and agile environment that embraces AI and cutting-edge technologies. Our culture fosters teamwork and progressive thinking, while our internal mobility program offers diverse career opportunities.
Leverage your expertise to lead technical delivery in the BICC Solution Centre, ensuring we deliver on our brand promise and drive organizational success together with our wide-reaching flight operation teams around the world.
Key Responsibilities
- Capture and communicate business direction, challenges, and requirements.
- Design, build and support large-scale batch and real-time data pipelines with data processing frameworks on Amazon Cloud Platform and hybrid cloud environments.
- Collaborate with application development teams, architects, Data Scientists and Machine Learning experts and stakeholders.
- Design, develop, implement, and support IT products and applications to meet the customer and IT needs.
- Provide an escalation point for solution analysts to deliver business solutions that include technology implementations and/or adaptations.
- Generate ideas and develop new approaches to improve overall system design and/or streamline delivery of product/system with adherence to security policy and data governance framework.
- Contribute to team resource planning and lead a team of Solution Analysts and vendor on implementation
- Keep abreast of the latest trends in technology, industry, and corresponding business domains
- Participate in Product Vertical Teams to continuously experiment, iterate, and deliver product objectives by adopting Agile development practices
Requirements
- Degree qualification in Information Technology related discipline, master’s preferred
- 8+ years of related IT experience, with over 5+ years of experience in the production deployment and lifecycle management of machine learning models. This includes skills in areas such as model deployment, monitoring, A/B testing, and operationalizing ML pipelines using cloud-based platforms like AWS SageMaker, Azure ML Studio, or similar tools. Expertise in this domain is a must.
- Proficient in machine learning frameworks essential for machine learning operation activities.
- Hands-on experience in implementing and managing continuous integration (CI) and continuous deployment (CD) pipelines for big data and machine learning applications using tools like Jenkins, GitLab CI/CD, etc.
- Familiarity with containerization and orchestration technologies, particularly Docker and Kubernetes, for deploying and managing big data and machine learning applications. Preferably using AWS EKS and OpenShift.
- Proficiency in infrastructure provisioning and configuration management using tools like Terraform, Ansible, AWS CDK, and AWS CloudFormation, etc.
- Experience in designing and maintaining data pipelines. Knowledge of PySpark, Python, Hadoop ecosystems, open-source development frameworks, etc.
- Experience in cloud-based design and development preferably using AWS.
- Experience in agile development processes and data-driven development.
Personal & Application Information
Cathay Pacific is an Equal Opportunities Employer. Personal data provided by job applicants will be used strictly in accordance with our Applicant Personal Information Collection Statement and for recruitment purposes only. Candidates not notified within eight weeks may consider their application unsuccessful. We keep records of your data for no longer than is necessary for the purpose for which we obtained them and any other permitted linked purposes. If your application is unsuccessful, we will keep your details on file for as long as is necessary to process your application or for the purposes of further job opportunities if you agree to such longer periods.
Key skills/competency
- Big Data
- Data Engineering
- Machine Learning Operations (MLOps)
- AWS
- Data Pipelines
- CI/CD
- Docker
- Kubernetes
- Terraform
- Agile Development
Skills & topics
- Solution Lead
- Big Data
- Data Engineering
- Machine Learning
- AWS
- Data Pipelines
- MLOps
- CI/CD
- Docker
- Kubernetes
- Terraform
- Agile
How to get hired
- Tailor your resume: Highlight your 8+ years of IT experience, focusing on 5+ years in ML model deployment and lifecycle management on cloud platforms.
- Showcase MLOps expertise: Emphasize your hands-on experience with CI/CD, Docker, Kubernetes, and infrastructure as code tools like Terraform.
- Demonstrate cloud proficiency: Detail your experience with AWS services, particularly for big data and ML pipelines.
- Prepare for technical interviews: Be ready to discuss your experience with PySpark, Python, Hadoop, and agile methodologies.
- Understand Cathay Pacific's culture: Research their commitment to AI, teamwork, and internal mobility to align your responses.
Technical preparation
Master AWS services for data and ML.,Practice PySpark and Python for data pipelines.,Set up CI/CD for ML model deployment.,Configure Docker and Kubernetes for applications.
Behavioral questions
Describe a complex data challenge you solved.,How do you ensure data governance and security?,Explain your approach to leading technical teams.,How do you stay updated on technology trends?
Frequently asked questions
- What are the key technical skills required for the Solution Lead – Big Data and Data Engineering role at Cathay Pacific?
- The Solution Lead role at Cathay Pacific requires extensive experience in big data and data engineering, with a strong focus on machine learning operations (MLOps). Key technical skills include designing and implementing large-scale data pipelines on AWS, managing ML model lifecycle (deployment, monitoring, A/B testing), proficiency in ML frameworks, hands-on experience with CI/CD tools (Jenkins, GitLab CI/CD), containerization (Docker, Kubernetes), infrastructure as code (Terraform, Ansible), and cloud development on AWS. Familiarity with PySpark, Python, and Hadoop ecosystems is also crucial.
- What level of experience is expected for the Solution Lead position at Cathay Pacific?
- Cathay Pacific expects a minimum of 8 years of related IT experience for this Solution Lead role. More specifically, over 5 years of experience must be dedicated to the production deployment and lifecycle management of machine learning models, including operationalizing ML pipelines on cloud platforms.
- Does Cathay Pacific offer opportunities for career growth within this Solution Lead role?
- Yes, Cathay Pacific emphasizes internal mobility and offers diverse career opportunities. The role itself is in a dynamic environment that fosters progressive thinking, and participation in Product Vertical Teams encourages continuous experimentation and growth within the company.
- What is the expected educational background for a Solution Lead at Cathay Pacific?
- A degree qualification in an Information Technology-related discipline is required. A Master's degree in a relevant field is preferred for the Solution Lead position at Cathay Pacific.
- How does Cathay Pacific approach agile development for their data and ML projects?
- Cathay Pacific adopts Agile development practices and data-driven development approaches. Solution Leads are expected to participate in Product Vertical Teams to continuously experiment, iterate, and deliver product objectives within this agile framework.
- What cloud platforms are utilized for the Solution Lead role at Cathay Pacific?
- The Solution Lead will primarily work with Amazon Cloud Platform (AWS) for designing, building, and supporting data pipelines and ML infrastructure. Experience with hybrid cloud environments is also expected, and specific AWS services like SageMaker, EKS, CloudFormation, and CDK are relevant.