Principal Databricks Data Engineer
Experience Required: 12–18 Years
Key Requirements
• 12–18 years of overall Data Engineering experience
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
• 5+ years of hands-on experience with Databricks and Apache Spark at scale
• Strong experience modernizing legacy Cloudera platforms including:
• CDH/CDP
• Hive
• HBase
• Impala
• Spark
• Modernize Cloudera platforms to Databricks Lakehouse architecture
• Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
• Ensure data reconciliation, audit integrity, and consistency during migration
• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
• Implement layered data architecture including:
• Raw / Landing Layer
• Curated / Conformed Layer
• Semantic / Consumption Layer
• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
• Strong experience with finance and risk data models including:
• General Ledger
• Sub-ledger
• Financial Hierarchies
• Credit Risk
• Liquidity Risk
• Market Risk
• Enable reporting capabilities including:
• Aggregation
• Drill-down
• Drill-back
• Build and manage semantic and consumption layers for BI, reporting, and analytics
• Define business metrics, dimensions, hierarchies, and KPIs
• Experience with:
• Databricks SQL
• Delta Tables
• dbt or similar frameworks
• Develop and optimize large-scale data pipelines using:
• PySpark
• Spark SQL
• Delta Lake
• Implement Medallion Architecture including:
• Bronze Layer
• Silver Layer
• Gold Layer
• Optimize workloads using:
• Z-ORDER
• OPTIMIZE
• Caching
• Cluster Configuration Tuning
• Implement:
• Data Governance
• Data Quality Frameworks
• Reconciliation Controls
• Exception Handling
• Establish data lineage and metadata management
• Ensure data security, access control, and compliance standards
• Experience with AWS or Azure cloud platforms
• Experience with CI/CD pipelines using:
• Git
• Terraform
• Jenkins
• Azure DevOps
• Familiarity with:
• Apache Airflow
• Databricks Workflows
• Experience with dbt is an advantage
• Act as a technical authority and lead enterprise architecture decisions
• Mentor senior engineers and establish engineering standards
• Collaborate with finance, risk, analytics, and governance stakeholders
• Translate complex data structures into business-ready insights
Nice-to-Have Skills
• Experience in BFSI, Capital Markets, or Regulatory Reporting
• Exposure to:
• SAP Finance
• Oracle Financials
• SAP S/4HANA
• Experience supporting AI/ML workloads
• Databricks and Cloud Certifications
Key Responsibilities
• Lead Cloudera to Databricks transformation initiatives
• Shape enterprise finance and risk data platforms
• Support regulatory, management, and analytical reporting systems
Essential Skills
• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Lakehouse Architecture
• Cloudera (CDH/CDP)
• Hive
• HBase
• Impala
• Data Warehouse & Data Lake
• Medallion Architecture
• Databricks SQL
• dbt
• AWS / Azure
• Airflow / Databricks Workflows
• CI/CD
• Terraform
• Jenkins
• Git
• Finance & Risk Data Models
• Enterprise Data Architecture
Principal Databricks Data Engineer • Toronto, ON, ca
We are seeking an experienced Principal Software Engineer (Data) to join our fast-paced, collaborative data team.In this role, you will have broad authority to drive the direction of our technograp... Show more
A leading data engineering firm in Toronto is seeking a Data Engineer with 2-5 years of experience in technologies such as PySpark, SQL, and Databricks.The role involves working in the Data Science... Show more
This Data Engineer role is a one-year contract, working on a hybrid in-office/remote model in the GTA.The compensation range is $60-$100/hr.Build and maintain ETL pipelines for enterprise data inte... Show more
Venture outside the ordinary - TMX Careers The TMX group of companies includes leading global exchanges such as the Toronto Stock Exchange, Montreal Exchange, and numerous innovative organizations ... Show more
The Principal Data Engineer will lead teams, design solutions, and deliver technology strategies for clients, requiring extensive software engineering experience. Show more
Director Data and AI, Embedded Finance.The primary focus of this role will be to design, build, and maintain the EMFI Data Application Platform, a financial analytics application.The incumbent will... Show more
At IBM, work is more than a job – it’s a calling: To build.To think along with clients and sell.Not just to do something better, but to attempt things you’ve never thought possible.Are you ready to... Show more
As a Data Engineer and Analytic you will be tasked with empowering internal stakeholders to delve deep into the complexities of the data in our warehouse.You will work closely with Product, Enginee... Show more
Fitch Group is currently seeking a Associate Director/Lead Data Engineer based out of our Toronto office.As a leading, global financial information services provider, Fitch Group delivers vital cre... Show more
Senior technical leadership role for an Enterprise Data & Analytics function, focusing on data solutions, setting standards, and hands-on delivery.Requires expertise in modern data stacks and mento... Show more
Get AI-powered advice on this job and more exclusive features.Direct message the job poster from Compunnel Inc.University degree in Computer Engineering or Computer Science.Solid grasp/experience w... Show more
Location : Preference to have candidates based in Toronto, next best, candidates would be located in Vancouver.Hybrid work environment of 1 - 2 days onsite.On behalf of our client, Affinity is seek... Show more
A leading IT consultancy is seeking a Mid-Senior Level Data Engineer for a contract role in Toronto.Candidates will design, develop, and maintain scalable data pipelines, ensuring data quality and ... Show more
A technology company is seeking an experienced Data Engineering Lead to guide technical direction for critical initiatives in data engineering.This fully remote position requires 10+ years of exper... Show more
Lead the architectural advancements of Agentic AI as a Principal Engineer.Oversee innovative projects in reinforcement learning and multi-agent systems to create scalable solutions.You will act as ... Show more
Tangerine is Canada’s leading direct bank.We offer flexible and accessible banking options, innovative products, and award-winning client service.Our employees come to work each day to help Canadia... Show more
Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Tiger Analytics is a global AI and analytics consulting firm.With data and technology at the core of o... Show more
Transform data into insights as an Advanced Data Engineer specializing in complex ETL pipelines.Embrace this remote opportunity to use your Python or Java and SQL skills effectively.This role invit... Show more
Press Tab to Move to Skip to Content Link.Select how often (in days) to receive an alert:.Ofc of Vice Pres, Research & Innovation.The Vice-President, Research & Innovation (VPRI) portfolio supports... Show more
Take charge as Lead Manager for the Data Engineering team, fostering growth and innovation within data projects.We seek a dedicated professional focused on customer-centric solutions and team succe... Show more
