We are looking for a skilled and motivated Data Engineer to join our team supporting Commonwealth Bank of Australia (CBA). The ideal candidate will have hands-on experience in designing, developing, and optimizing scalable data pipelines using modern big data technologies and cloud platforms. You will work closely with cross-functional teams to build reliable data solutions that support business intelligence, analytics, and regulatory reporting.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Develop and optimize Spark and Scala applications for large-scale data processing.
• Build and manage data solutions using Hadoop technologies including HDFS and Hive.
• Design, implement, and optimize Snowflake data warehouse solutions.
• Write high-performance SQL queries and perform query optimization.
• Develop cloud-native data pipelines using AWS services such as S3, Glue, Athena, and Redshift.
• Perform data validation, quality checks, and troubleshooting of production issues.
• Collaborate with business analysts, architects, and development teams to understand data requirements.
• Manage source code using GitHub and automate deployments through GitHub Actions and TeamCity.
• Create and maintain Shell scripts for automation and operational support.
• Participate in release management, production support, and continuous improvement initiatives.
• Follow banking security, governance, and compliance standards.

Required Skills

• Strong experience with Spark and Scala
• Hands-on experience with Hadoop (HDFS, Hive)
• Experience working with Snowflake
• Advanced SQL skills including performance tuning and query optimization
• Experience with AWS services:
  • Amazon S3
  • AWS Glue
  • Amazon Athena
  • Amazon Redshift
• Experience in ETL/Data Pipeline development
• Version control using GitHub
• CI/CD using GitHub Actions and TeamCity
• Shell Scripting
• Experience working in the Banking domain

Preferred Skills

• Experience in Financial Crime, AML, or Fraud Analytics
• AWS Certification
• Oracle Database
• Scheduling tools such as AutoSys or Apache Airflow
• Production Support and Release Management experience

Required Experience

• 3–5 years of experience as a Data Engineer or Big Data Engineer.
• Strong understanding of data engineering best practices and cloud-based data platforms.
• Excellent analytical, troubleshooting, and communication skills.