Senior Data Engineer (AWS Cloud)
Sydney, NSW - CBP North, 1 Harbour Street
Commonwealth Bank
CommBank offers personal banking, business solutions, institutional banking, company information, and moreSenior Data Engineer (AWS Cloud)
You are passionate about modernising the HR ecosystem through strategic and secure management of cloud data products, pipelines and databases
We are undergoing one of Australia’s largest digital transformations and establishing an Engineering function within HR.
Together we bring an engineering culture and passion for DevSecOps and GitOps within the cloud
Do work that matters
As a Senior Data Engineer and database specialist you will be designing, creating and managing the cloud databases and data pipelines that underpin our decoupled cloud architecture and API first approach. You have proven expertise in database design, data ingestion, transformation, data writing, scheduling and query management within a cloud environment.
You will have proven experience and expertise in working with AWS Cloud Infrastructure Engineers, Software/API Developers and Architects to design, develop, deploy and operate data services and solutions that underpin a cloud ecosystem. You will take ownership and accountability of functional and non-functional design and work within a team of Engineers to create innovative solutions that unlock value and modernise technology designs.
You will role model continuous improvement mindset in the team, and in your project interactions, by taking technical ownership of key assets, including roadmaps and technical direction of data services running on our AWS environments.
See yourself in our team
The HR Technology function works in an Agile methodology with our HR business to plan, prioritise and deliver on high value HR & technology objectives with key results that can be experienced by all of our employees.
You will work within the Middleware, ETL and Storage Chapter that is working on initiatives such as establishing decoupled HR architecture through API’s and establishing ML Ops (Sagemaker). You will do this through expert knowledge of database design, data engineering and data transformation. A typical day would work with data producing systems such as Workday Engineers to define, extract, transform and load to PostgresDB and then working with AWS Engineers and API developers to expose the data products through API’s for secure consumption in various HR Applications and solutions.
We’re interested in hearing from people who:
Can design and implement databases for data integration in the enterprise
Are developing and maintain scripts or ETL (Extract, Transform, Load) processes to ingest raw data from various sources, including JSON and XML files
Can design and implement transformation logic to convert raw JSON and XML data into structured formats such as CSV to write to databases
Can create and optimize scripts to write transformed data into relational databases (e.g., PostgreSQL- Ensuring that data is properly indexed, normalized, and stored efficiently.
Can automate data ingestion and transformation processes using scheduling tools.
Monitor and troubleshoot data pipelines to ensure reliability and performance.Have experience working through performance and scaling through horizontal scaling designs vs database tuning
Can design application logical database requirements and implement physical solutions
Can develop back-end database code to support API requirements
Can collaborate with business and technical teams in order to design and build critical databases and data pipelines
Can performance tune applications from a database code and design perspective
Can advise business owners on strategic database direction and application solution design
Tech skills
We use a broad range of tools, languages, and frameworks. We don’t expect you to know them all but having significant experience and exposure with some of these (or equivalents) will set you up for success in this team.
AWS Data products such as AWS Glue and AWS EMR
AWS Aurora RDS such as PostgreSQL
AWS S3 ingestion, transformation and writing to databases
Realtime streaming technologies , Kafka
Proficiency in programming languages like Python, Java, or Ruby for developing data ingestion and transformation scripts.
Strong knowledge of SQL for writing, optimizing, and debugging queries.
Familiarity with database design, indexing, and normalization principles.
Understanding of data formats (JSON, CSV, XML) and techniques for converting between them. Ability to handle data validation, cleaning, and transformation.Proficiency in automation tools and scripting (e.g., bash scripting, cron jobs) for scheduling and monitoring data processes.
Experience with version control systems (e.g., Git) for managing code and collaboration.
Working with us:
Whether you’re passionate about customer service, driven by data, or called by creativity, a career with CommBank is for you.
Our people bring their diverse backgrounds and unique perspectives to build a respectful, inclusive, and flexible workplace with flexible work locations. One where we’re driven by our values, and supported to share ideas, initiatives, and energy. One where making a positive impact for customers, communities and each other is part of our every day.
Here, you’ll thrive. You’ll be supported when faced with challenges and empowered to tackle new opportunities. We’re hiring engineers from across all of Australia and have opened technology hubs in Melbourne and Perth. We really love working here, and we think you will too.
We support our people with the flexibility to balance where work is done with at least half their time each month connecting in office. We also have many other flexible working options available including changing start and finish times, part-time arrangements and job share to name a few. Talk to us about how these arrangements might work in the role you’re interested in.
If this sounds like the role for you then we would love to hear from you. Apply today!
If you're already part of the Commonwealth Bank Group (including Bankwest, x15ventures), you'll need to apply through Sidekick to submit a valid application. We’re keen to support you with the next step in your career.
We're aware of some accessibility issues on this site, particularly for screen reader users. We want to make finding your dream job as easy as possible, so if you require additional support please contact HR Direct on 1800 989 696.
Advertising End Date: 20/07/2024* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Agile APIs Architecture AWS AWS Glue CSV Data pipelines Engineering ETL Git Java JSON Kafka Machine Learning Pipelines PostgreSQL Python RDBMS Ruby SageMaker SQL Streaming XML
Perks/benefits: Equity / stock options Flex hours Startup environment
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Research Scientist jobs
- Open Data Science Manager jobs
- Open Junior Data Analyst jobs
- Open Business Data Analyst jobs
- Open Data Scientist II jobs
- Open Principal Data Scientist jobs
- Open BI Analyst jobs
- Open Sr Data Engineer jobs
- Open Business Intelligence Engineer jobs
- Open Data Science Intern jobs
- Open Senior Business Intelligence Analyst jobs
- Open Lead Data Analyst jobs
- Open Sr. Data Scientist jobs
- Open Azure Data Engineer jobs
- Open Software Engineer, Machine Learning jobs
- Open Junior Data Scientist jobs
- Open MLOps Engineer jobs
- Open Manager, Data Engineering jobs
- Open Marketing Data Analyst jobs
- Open Data Analytics Engineer jobs
- Open Data Engineer III jobs
- Open Junior Data Engineer jobs
- Open Data Engineering Manager jobs
- Open Product Data Analyst jobs
- Open Data Analyst II jobs
- Open Privacy-related jobs
- Open GCP-related jobs
- Open Tableau-related jobs
- Open Excel-related jobs
- Open ML models-related jobs
- Open Data pipelines-related jobs
- Open APIs-related jobs
- Open PhD-related jobs
- Open PyTorch-related jobs
- Open Finance-related jobs
- Open LLMs-related jobs
- Open TensorFlow-related jobs
- Open Deep Learning-related jobs
- Open Data visualization-related jobs
- Open Consulting-related jobs
- Open Business Intelligence-related jobs
- Open Generative AI-related jobs
- Open Data governance-related jobs
- Open NLP-related jobs
- Open CI/CD-related jobs
- Open DevOps-related jobs
- Open Kubernetes-related jobs
- Open Git-related jobs
- Open Docker-related jobs
- Open Snowflake-related jobs