Associate Director, Principal Data Engineer

Job Description

What is the opportunity? As a Data Engineering Solution Architect, you will join a highly talented team at RBC Capital Markets responsible for designing and delivering next-generation data engineering platforms on Databricks. You will be the hands-on technical anchor driving Proof of Concepts (POCs) through to full production delivery, working across large-scale Spark-based data pipelines, ML engineering workflows, and Delta Lake architectures.You will bring deep, practitioner-level expertise in Databricks and Apache Spark — not just designing solutions on whiteboards, but rolling up your sleeves to build, tune, and deliver them. You will work closely with data scientists, ML engineers, business analysts, and platform teams to architect and implement scalable, high-performance data solutions that power critical financial workflows across RBC.You will exhibit a strong engineering mindset, display technical leadership, and bring the energy and rigor needed to raise the bar on data engineering standards across the organization.What will you do?

• Lead the end-to-end architecture, design, and hands-on delivery of data engineering solutions on the Databricks Lakehouse Platform, from POC through to production.
• Drive Spark code development, optimization, and fine-tuning to ensure high-performance, cost-efficient data pipelines at scale.
• Architect and implement Delta Lake solutions including schema design, medallion architecture, data quality frameworks, and incremental ingestion patterns.
• Champion best practices in Databricks Workflows, Unity Catalog, Auto Loader, and structured streaming for both batch and real-time data processing.
• Design and build ML Engineering pipelines using MLflow, Feature Store, and Model Serving within the Databricks ecosystem.
• Collaborate closely with product owners, business analysts, data scientists, and platform engineers to translate business requirements into robust technical solutions.
• Conduct performance benchmarking and tuning of Spark jobs, cluster configurations, and storage layouts to optimize cost and runtime.
• Establish and enforce coding standards, peer review practices, and CI/CD pipelines for data engineering workloads.
• Proactively evaluate emerging Databricks and Apache Spark capabilities and assess their applicability to current and future RBC use cases.
• Provide technical mentorship and hands-on guidance to junior and mid-level data engineers across the team.

What do you need to succeed?Must-have

• 8+ years of hands-on, production-grade experience with Apache Spark and Databricks, including Spark code development and fine-tuning.
• Databricks Certified Data Engineer Associate/Professional — mandatory pre-requisite.
• Databricks Certified Machine Learning Associate/Professional — mandatory pre-requisite.
• Proven track record of leading and delivering end-to-end data engineering solutions on Databricks in a financial services or similarly complex enterprise environment.
• Deep expertise in Delta Lake, Delta Live Tables, Unity Catalog, and the Databricks Lakehouse architecture.
• Strong proficiency in Python (PySpark) and/or Scala for Spark development, with demonstrable experience in performance tuning (partitioning, caching, shuffle optimization, adaptive query execution).
• Experience architecting and delivering POCs independently — from scoping, prototyping, to stakeholder-ready demonstration.
• Hands-on experience with cloud platforms (Azure or AWS) for big data workloads, including cloud storage, networking, and IAM.
• Solid understanding of MLflow lifecycle management, model versioning, and ML pipeline orchestration within Databricks.
• Familiarity with CI/CD tooling for data pipelines — including GitHub Actions, Azure DevOps, or equivalent.

Nice-to-have

• Experience with real-time streaming architectures using Kafka, Spark Structured Streaming, or Delta Live Tables.
• Exposure to Databricks on Azure (ADLS Gen2, Azure Data Factory integration) or on AWS (S3, Glue integration).
• Experience with infrastructure-as-code tooling such as Terraform for Databricks workspace provisioning and cluster management.
• Familiarity with Agile/Scrum delivery methodologies.
• Knowledge of data governance frameworks, data cataloguing tools, and data quality standards in a regulated financial environment.
• Additional Databricks certifications (e.g. Databricks Certified Associate Developer for Apache Spark) are a plus.

Job SkillsBig Data Management, Cloud Computing, Database Development, Data Engineering, Data Mining, Data Pipelines, Data Warehousing (DW), Design, End-to-End Testing, ETL Processing, Group Problem Solving, MLflow, Quality Management, Real Time Data, Technical SolutionsAdditional Job Details

Address:

RBC CENTRE, 155 WELLINGTON ST W:TORONTO

City:

Toronto

Country:

Canada

Work hours/week:

37.5

Employment Type:

Full time

Platform:

CAPITAL MARKETS

Job Type:

Regular

Pay Type:

Salaried

Posted Date:

2026-06-05

Application Deadline:

2026-08-14Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date aboveOur Employment OpportunitiesAt RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.

Join our Talent Community

Stay in-the-know about great career opportunities at RBC. Sign up and get customized info on our latest jobs, career tips and Recruitment events that matter to you.

Expand your limits and create a new future together at RBC. Find out how we use our passion and drive to enhance the well-being of our clients and communities at jobs.rbc.com.

RBC is presently inviting candidates to apply for this existing vacancy. Applying to this posting allows you to express your interest in this current career opportunity at RBC. Qualified applicants may be contacted to review their resume in more detail.