Key responsibilities
1. Design the Collections Snowflake and PySpark architecture.
2. Lead integration with commbank.data.
3. Model HPS card and account data for Collections use cases.
4. Define data contracts and schema-evolution standards.
5. Establish automated data-quality and reconciliation controls.
6. Design restartable, idempotent and recoverable pipelines.
7. Optimise Snowflake and PySpark workloads.
8. Establish production monitoring and data-pipeline observability.
9. Lead technical reviews and complex data investigations.
10. Mentor Senior Data Engineers and Data Engineers.
11. Work closely with Cards MOD, PCC and platform teams.
Essential skills
a) Strong production Snowflake experience.
b) Strong production PySpark / Spark experience.
c) Python.
d) Advanced SQL.
e) AWS.
f) S3.
g) Enterprise data engineering.
h) ETL/ELT and data modelling.
i) Data quality and reconciliation.
j) CI/CD and automated testing.
k) Production support.
Highly regarded
EMR / EMR Serverless.
Step Functions.
Lambda.
Cards or payments data.
Collections or lending.
PowerCurve Collections.