Data Engineer

Typical Day in Role

• Designing, building, operationalizing the Wealth Data Hub (WDH) using Google Cloud Platform (GCP) data services such as DataProc, Dataflow, CloudSQL, BigQuery, CloudSpanner in combination with third parties such as Spark, Apache Beam / composer, DBT, Cloud Pub / Sub, Confluent Kafka, Cloud storage Cloud Functions & Github
• Designing and implementing data ingestion patterns that will support batch, streaming and API interface on both the Ingress and Egress.
• Guide a team of data engineers and work hands on in developing framework and custom code using best practices that will meet the demanding performance requirements
• Take a lead in designing and building production data pipelines from data ingestion to consumption using GCP services, Java, Python, Scala, BigQuery, DBT, SQL etc.
• Experience using Cloud Dataflow using Java / Python for deploying streaming jobs in GCP as well as batch jobs using text / JSON files and writing them to BigQuery
• Building and managing data pipelines with a deep understanding of workflow orchestration, task scheduling and dependency management
• Ability to do proof of technology using GCP technologies and work with data architects, solution architects to achieve the desired results and performance.
• Provide end-to-end technical guidance and expertise on how to effectively use Google Cloud to build solutions; creatively applying cloud infrastructure and platform services to help solve business problems; and communicating these approaches to different business users
• Provide guidance on Implementing application logging, notification, jobs monitoring and performance monitoring

Candidate Requirements / Must Have Skills

Nice-To-Have Skills

Education

#J-18808-Ljbffr