Talent.com
Astra North Infoteck Inc.
Principal Databricks Data EngineerAstra North Infoteck Inc. • Toronto, ON, ca
Principal Databricks Data Engineer

Principal Databricks Data Engineer

Astra North Infoteck Inc. • Toronto, ON, ca
Il y a 9 jours
Type de contrat
  • Temps plein
Description de poste

Principal Databricks Data Engineer

Experience Required: 12–18 Years

Key Requirements

• 12–18 years of overall Data Engineering experience
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
• 5+ years of hands-on experience with Databricks and Apache Spark at scale
• Strong experience modernizing legacy Cloudera platforms including:
• CDH/CDP
• Hive
• HBase
• Impala
• Spark
• Modernize Cloudera platforms to Databricks Lakehouse architecture
• Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
• Ensure data reconciliation, audit integrity, and consistency during migration
• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
• Implement layered data architecture including:
• Raw / Landing Layer
• Curated / Conformed Layer
• Semantic / Consumption Layer
• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
• Strong experience with finance and risk data models including:
• General Ledger
• Sub-ledger
• Financial Hierarchies
• Credit Risk
• Liquidity Risk
• Market Risk
• Enable reporting capabilities including:
• Aggregation
• Drill-down
• Drill-back
• Build and manage semantic and consumption layers for BI, reporting, and analytics
• Define business metrics, dimensions, hierarchies, and KPIs
• Experience with:
• Databricks SQL
• Delta Tables
• dbt or similar frameworks
• Develop and optimize large-scale data pipelines using:
• PySpark
• Spark SQL
• Delta Lake
• Implement Medallion Architecture including:
• Bronze Layer
• Silver Layer
• Gold Layer
• Optimize workloads using:
• Z-ORDER
• OPTIMIZE
• Caching
• Cluster Configuration Tuning
• Implement:
• Data Governance
• Data Quality Frameworks
• Reconciliation Controls
• Exception Handling
• Establish data lineage and metadata management
• Ensure data security, access control, and compliance standards
• Experience with AWS or Azure cloud platforms
• Experience with CI/CD pipelines using:
• Git
• Terraform
• Jenkins
• Azure DevOps
• Familiarity with:
• Apache Airflow
• Databricks Workflows
• Experience with dbt is an advantage
• Act as a technical authority and lead enterprise architecture decisions
• Mentor senior engineers and establish engineering standards
• Collaborate with finance, risk, analytics, and governance stakeholders
• Translate complex data structures into business-ready insights

Nice-to-Have Skills

• Experience in BFSI, Capital Markets, or Regulatory Reporting
• Exposure to:
• SAP Finance
• Oracle Financials
• SAP S/4HANA
• Experience supporting AI/ML workloads
• Databricks and Cloud Certifications

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives
• Shape enterprise finance and risk data platforms
• Support regulatory, management, and analytical reporting systems

Essential Skills

• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Lakehouse Architecture
• Cloudera (CDH/CDP)
• Hive
• HBase
• Impala
• Data Warehouse & Data Lake
• Medallion Architecture
• Databricks SQL
• dbt
• AWS / Azure
• Airflow / Databricks Workflows
• CI/CD
• Terraform
• Jenkins
• Git
• Finance & Risk Data Models
• Enterprise Data Architecture







Créer une alerte emploi pour cette recherche

Principal Databricks Data Engineer • Toronto, ON, ca

Offres similaires

Principal Data Engineer (Azure)

Tiger AnalyticsToronto, ON, CA
Temps plein

Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Tiger Analytics is a global AI and analytics consulting firm.With data and technology at the core of o... Voir plus

 • Offre sponsorisée

Principal Software Engineer - Data

ZoomInfoToronto
Temps plein

We are seeking an experienced Principal Software Engineer (Data) to join our fast-paced, collaborative data team.In this role, you will have broad authority to drive the direction of our technograp... Voir plus

 • Offre sponsorisée

Big Data Engineer

TMX GroupToronto, ON, CA
Temps plein

We are seeking a seasoned and motivated Big Data Engineer to join our growing Enterprise Data and Advanced Analytics Platform team.The ideal candidate is an expert data pipeline builder and data wr... Voir plus

 • Offre sponsorisée

Senior Data Platform Engineer

EPAM SystemsToronto, Ontario, Canada
Temps plein

Python background to join our team.This role is ideal for a data engineering professional experienced in delivering robust data platforms and implementing data fabric solutions.As a Senior Data Pla... Voir plus

 • Offre sponsorisée

Principal Data Engineer

RAVLToronto
Temps plein

RAVL helps technologists accelerate their careers.At RAVL, we connect strategy with execution, care deeply about the people we work with, and measure success by the lasting impact we leave behind.O... Voir plus

 • Offre sponsorisée

Principal Data Engineer

AvisoToronto, Ontario, Canada
Temps plein

We are looking for a Principal Data Engineer to join our Data and Analytics Group within the Technology Services Partners (TSP) department.Reporting to the Sr Director, Data Platform Delivery and E... Voir plus

 • Offre sponsorisée

Senior Data Engineer - Databricks & Azure for Banking

Soar ConsultantsToronto, ON, CA
Temps plein

A leading data consulting firm is seeking a Data Engineer for a client in Toronto.The role requires experience with Azure, Databricks, and strong skills in data analysis and integration methodologi... Voir plus

 • Offre sponsorisée

Data Engineer

LeadingtalentMarkham, ON, CA
Temps plein

At IBM, work is more than a job – it’s a calling: To build.To think along with clients and sell.Not just to do something better, but to attempt things you’ve never thought possible.Are you ready to... Voir plus

 • Offre sponsorisée

Oliver Wyman - Principal Data Engineer - $240,000 - $260,000 A Year

Oliver WymanToronto, Canada
Temps plein

The Principal Data Engineer will lead teams, design solutions, and deliver technology strategies for clients, requiring extensive software engineering experience. Voir plus

 • Offre sponsorisée

Junior Databricks Data Engineer — Hybrid Lakehouse - C$70,000 - C$94,000 A Year

The Citco GroupToronto, Canada
Temps plein

Junior Data Engineer needed for a financial services firm in Toronto to build and manage data pipelines on Databricks, optimize ETL/ELT, and ensure data governance in a hybrid setting. Voir plus

 • Offre sponsorisée

Senior Data Engineer ( Databricks/Snowflake)

TEEMA Solutions GroupToronto, ON, CA
Temps plein

Toronto (3 days downtown/Hybrid).Our client is interviewing to onboard a Senior Data Engineers to join their Data & Analytics team within the Wealth Technology group.We are looking for Data Enginee... Voir plus

 • Offre sponsorisée

Data Engineer: PySpark, Databricks & Tableau

Highbrow LLCToronto, ON, CA
Temporaire

A leading data engineering firm in Toronto is seeking a Data Engineer with 2-5 years of experience in technologies such as PySpark, SQL, and Databricks.The role involves working in the Data Science... Voir plus

 • Offre sponsorisée

Data Engineer

Inizio Partners CorpToronto
Temps plein

Job Title: Consultant Data Engineer.We're looking for a Senior Data Engineer with strong data platform experience to help with support of existing AWS/dbt platform and that can help with standing u... Voir plus

 • Offre sponsorisée

Lead Data Engineer – Databricks

TEEMAToronto, Ontario, Canada
Temps plein

Team Lead, Data Engineering – Databricks.Our client has provided specialist financial services to alternative investment funds, investors, multinationals, and private clients worldwide.Our data eng... Voir plus

 • Offre sponsorisée

Senior AWS Databricks Data Engineer - Hybrid Toronto

AkkodisToronto, ON, CA
Temps plein

A leading IT consultancy is seeking a Mid-Senior Level Data Engineer for a contract role in Toronto.Candidates will design, develop, and maintain scalable data pipelines, ensuring data quality and ... Voir plus

 • Offre sponsorisée

Senior Data Engineer: Databricks Lakehouse & ML Pipelines

TeraWatt InfrastructureToronto
Temps plein

A leading infrastructure company in Toronto is seeking a Senior Data Engineer to design and implement scalable data architectures.In this role, you will work with cross-functional teams to optimize... Voir plus

 • Offre sponsorisée

Senior Databricks Data Engineer — Lakehouse & Ai Pipelines - C$120,000 - C$150,000 A Year

The Citco Group LimitedToronto, Canada
Temps plein

A leading financial services provider in Toronto seeks a Senior Data Engineer to advance its data initiatives.This role involves designing and optimizing data pipelines on the Databricks Lakehouse ... Voir plus

 • Offre sponsorisée

Advanced Data Engineer for Large Datasets and ETL Pipelines

LumenaltaToronto, ON, CA
Temps plein

Transform data into insights as an Advanced Data Engineer specializing in complex ETL pipelines.Embrace this remote opportunity to use your Python or Java and SQL skills effectively.This role invit... Voir plus

 • Offre sponsorisée

Lead Data Engineer

Thomson ReutersToronto, ON, CA
Temps plein

We are seeking a highly skilled and strategic Lead Data Engineer to join our team.In this advanced individual contributor role, you will spearhead the development of sophisticated, insightful, and ... Voir plus

 • Offre sponsorisée

Senior Data Engineer — Aws/Databricks Data Platform Lead - $90,000 - $140,000 A Year

Vanguard CanadaToronto, Canada
Temps plein

A leading financial services firm in Toronto seeks a Senior Data Engineer to design and optimize data pipelines utilizing AWS and Databricks.The role requires 5+ years of experience in data enginee... Voir plus