Talent.com
Astra North Infoteck Inc.
Principal Databricks Data EngineerAstra North Infoteck Inc. • Toronto, ON, ca
Principal Databricks Data Engineer

Principal Databricks Data Engineer

Astra North Infoteck Inc. • Toronto, ON, ca
24 days ago
Job type
  • Full-time
Job description
Job Description

Principal Databricks Data Engineer

Experience Required: 12–18 Years

Key Requirements

• 12–18 years of overall Data Engineering experience
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
• 5+ years of hands-on experience with Databricks and Apache Spark at scale
• Strong experience modernizing legacy Cloudera platforms including:
• CDH/CDP
• Hive
• HBase
• Impala
• Spark
• Modernize Cloudera platforms to Databricks Lakehouse architecture
• Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
• Ensure data reconciliation, audit integrity, and consistency during migration
• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
• Implement layered data architecture including:
• Raw / Landing Layer
• Curated / Conformed Layer
• Semantic / Consumption Layer
• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
• Strong experience with finance and risk data models including:
• General Ledger
• Sub-ledger
• Financial Hierarchies
• Credit Risk
• Liquidity Risk
• Market Risk
• Enable reporting capabilities including:
• Aggregation
• Drill-down
• Drill-back
• Build and manage semantic and consumption layers for BI, reporting, and analytics
• Define business metrics, dimensions, hierarchies, and KPIs
• Experience with:
• Databricks SQL
• Delta Tables
• dbt or similar frameworks
• Develop and optimize large-scale data pipelines using:
• PySpark
• Spark SQL
• Delta Lake
• Implement Medallion Architecture including:
• Bronze Layer
• Silver Layer
• Gold Layer
• Optimize workloads using:
• Z-ORDER
• OPTIMIZE
• Caching
• Cluster Configuration Tuning
• Implement:
• Data Governance
• Data Quality Frameworks
• Reconciliation Controls
• Exception Handling
• Establish data lineage and metadata management
• Ensure data security, access control, and compliance standards
• Experience with AWS or Azure cloud platforms
• Experience with CI/CD pipelines using:
• Git
• Terraform
• Jenkins
• Azure DevOps
• Familiarity with:
• Apache Airflow
• Databricks Workflows
• Experience with dbt is an advantage
• Act as a technical authority and lead enterprise architecture decisions
• Mentor senior engineers and establish engineering standards
• Collaborate with finance, risk, analytics, and governance stakeholders
• Translate complex data structures into business-ready insights

Nice-to-Have Skills

• Experience in BFSI, Capital Markets, or Regulatory Reporting
• Exposure to:
• SAP Finance
• Oracle Financials
• SAP S/4HANA
• Experience supporting AI/ML workloads
• Databricks and Cloud Certifications

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives
• Shape enterprise finance and risk data platforms
• Support regulatory, management, and analytical reporting systems

Essential Skills

• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Lakehouse Architecture
• Cloudera (CDH/CDP)
• Hive
• HBase
• Impala
• Data Warehouse & Data Lake
• Medallion Architecture
• Databricks SQL
• dbt
• AWS / Azure
• Airflow / Databricks Workflows
• CI/CD
• Terraform
• Jenkins
• Git
• Finance & Risk Data Models
• Enterprise Data Architecture








Requirements
Sailpoint
Create a job alert for this search

Principal Databricks Data Engineer • Toronto, ON, ca

Similar jobs

Principal Data Engineer (Azure)

Tiger AnalyticsToronto, ON, CA
Full-time

Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Tiger Analytics is a global AI and analytics consulting firm.With data and technology at the core of o... Show more

 • Promoted

Principal Software Engineer - Data

ZoomInfoToronto
Full-time

We are seeking an experienced Principal Software Engineer (Data) to join our fast-paced, collaborative data team.In this role, you will have broad authority to drive the direction of our technograp... Show more

 • Promoted

Principal Engineer — Ai-Driven Data Extraction Platform - C$202,000 - C$278,000 A Year

Leading Data Intelligence FirmToronto County, Canada
Full-time

Lead the design and reliability of a global financial data extraction platform, focusing on AI-driven solutions and mentoring engineers. Show more

 • Promoted

Data Engineer: PySpark, Databricks & Tableau

Highbrow LLCToronto
Full-time +1

A leading data engineering firm in Toronto is seeking a Data Engineer with 2-5 years of experience in technologies such as PySpark, SQL, and Databricks.The role involves working in the Data Science... Show more

 • Promoted

Data Platform Engineer

Princeton IT Services, IncToronto, ON, CA
Full-time

Director Data and AI, Embedded Finance.The primary focus of this role will be to design, build, and maintain the EMFI Data Application Platform, a financial analytics application.The incumbent will... Show more

 • Promoted

Principal Data Engineer

Fitch RatingsToronto, ON, CA
Full-time

Fitch Group is currently seeking a Associate Director/Lead Data Engineer based out of our Toronto office.As a leading, global financial information services provider, Fitch Group delivers vital cre... Show more

 • Promoted

Oliver Wyman - Principal Data Engineer - $240,000 - $260,000 A Year

Oliver WymanToronto, Canada
Full-time

Principal Data Engineer role at Oliver Wyman.Lead technical teams, design cloud-native solutions, and mentor staff.Experience with software engineering, Agile methodologies, and communication skill... Show more

 • Promoted

Senior Databricks Data Engineer — Lakehouse & Ai Pipelines - C$120,000 - C$150,000 A Year

Financial Services ProviderNorth York, Canada
Full-time

Data Engineer needed in Toronto to design and optimize data pipelines on Databricks Lakehouse Platform using Python, Spark, and AWS for a financial services provider. Show more

 • Promoted

Senior Data Engineer ( Databricks/Snowflake)

TEEMA Solutions GroupToronto, ON, CA
Full-time

Toronto (3 days downtown/Hybrid).Our client is interviewing to onboard a Senior Data Engineers to join their Data & Analytics team within the Wealth Technology group.We are looking for Data Enginee... Show more

 • Promoted

Principal Data Engineer

AvisoToronto, ON, CA
Full-time

We are looking for a Principal Data Engineer to join our Data and Analytics Group within the Technology Services Partners (TSP) department.Reporting to the Sr Director, Data Platform Delivery and E... Show more

 • Promoted

Data Engineer

Inizio Partners CorpToronto
Full-time

Job Title: Consultant Data Engineer.We're looking for a Senior Data Engineer with strong data platform experience to help with support of existing AWS/dbt platform and that can help with standing u... Show more

 • Promoted

Senior Databricks Data Engineer - Lakehouse - C$120,000 - C$150,000 A Year

Citco GSGSToronto County, Canada
Full-time

Seeking a Senior Data Engineer with Databricks, Python, and AWS experience to optimize data pipelines. Show more

 • Promoted

Principal Engineer, Enterprise Data & Analytics - $120,000 - $150,000 A Year

QuadReal Property GroupNorth York, Canada
Full-time

Senior technical leader responsible for data and analytics solutions, setting standards, mentoring, and hands-on delivery of complex data problems for a global real estate investment company. Show more

 • Promoted

Senior AWS Databricks Data Engineer - Hybrid Toronto

AkkodisToronto, ON, CA
Full-time

A leading IT consultancy is seeking a Mid-Senior Level Data Engineer for a contract role in Toronto.Candidates will design, develop, and maintain scalable data pipelines, ensuring data quality and ... Show more

 • Promoted

Principal Data Engineer - Tangerine

Scotiabanktoronto, on, Canada
Full-time

Tangerine is Canada’s leading direct bank.We offer flexible and accessible banking options, innovative products, and award-winning client service.Our employees come to work each day to help Canadia... Show more

 • Promoted

Data Engineer

Adastra Corporationmarkham, on, Canada
Full-time

Adastra is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms that support advanced analytics, reporting, and AI initiatives.This r... Show more

 • Promoted

Databricks Engineer

PeoplePilotToronto, Ontario, Canada
Full-time

Data Warehouse and Data Lake platforms.Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.Redesign ingestion, transformation, a... Show more

 • Promoted

Principal Engineer — Ai-Driven Data Extraction Platform - C$202,000 - C$278,000 A Year

Data Intelligence FirmNorth York, Canada
Full-time

Lead the design and reliability of a global financial data extraction platform, focusing on AI-driven solutions and system architecture for a leading data intelligence firm. Show more

 • Promoted

Affinity Principal Engineer Data Solutions

AffinityToronto, Canada
Full-time

Principal Data Solutions Engineer at Affinity in Toronto, driving analytics engineering quality and innovation.Hands-on technical leadership role focused on data modeling, architecture, and mentoring. Show more

 • Promoted

Principal Data Solutions Engineer - C$120,000 - C$150,000 A Year

Leading Real Estate FirmNorth York, Canada
Full-time

Seeking a Principal Data Solutions Engineer for a leading real estate firm to lead data analytics solutions, provide mentorship, and develop technical standards using SQL and Python. Show more