Talent.com
Astra North Infoteck Inc.
Principal Databricks Data EngineerAstra North Infoteck Inc. • Toronto, ON, ca
Principal Databricks Data Engineer

Principal Databricks Data Engineer

Astra North Infoteck Inc. • Toronto, ON, ca
Il y a 3 jours
Type de contrat
  • Temps plein
  • Quick Apply
Description de poste
Job Description

Principal Databricks Data Engineer

Experience Required: 12–18 Years

Key Requirements

• 12–18 years of overall Data Engineering experience
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
• 5+ years of hands-on experience with Databricks and Apache Spark at scale
• Strong experience modernizing legacy Cloudera platforms including:
• CDH/CDP
• Hive
• HBase
• Impala
• Spark
• Modernize Cloudera platforms to Databricks Lakehouse architecture
• Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
• Ensure data reconciliation, audit integrity, and consistency during migration
• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
• Implement layered data architecture including:
• Raw / Landing Layer
• Curated / Conformed Layer
• Semantic / Consumption Layer
• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
• Strong experience with finance and risk data models including:
• General Ledger
• Sub-ledger
• Financial Hierarchies
• Credit Risk
• Liquidity Risk
• Market Risk
• Enable reporting capabilities including:
• Aggregation
• Drill-down
• Drill-back
• Build and manage semantic and consumption layers for BI, reporting, and analytics
• Define business metrics, dimensions, hierarchies, and KPIs
• Experience with:
• Databricks SQL
• Delta Tables
• dbt or similar frameworks
• Develop and optimize large-scale data pipelines using:
• PySpark
• Spark SQL
• Delta Lake
• Implement Medallion Architecture including:
• Bronze Layer
• Silver Layer
• Gold Layer
• Optimize workloads using:
• Z-ORDER
• OPTIMIZE
• Caching
• Cluster Configuration Tuning
• Implement:
• Data Governance
• Data Quality Frameworks
• Reconciliation Controls
• Exception Handling
• Establish data lineage and metadata management
• Ensure data security, access control, and compliance standards
• Experience with AWS or Azure cloud platforms
• Experience with CI/CD pipelines using:
• Git
• Terraform
• Jenkins
• Azure DevOps
• Familiarity with:
• Apache Airflow
• Databricks Workflows
• Experience with dbt is an advantage
• Act as a technical authority and lead enterprise architecture decisions
• Mentor senior engineers and establish engineering standards
• Collaborate with finance, risk, analytics, and governance stakeholders
• Translate complex data structures into business-ready insights

Nice-to-Have Skills

• Experience in BFSI, Capital Markets, or Regulatory Reporting
• Exposure to:
• SAP Finance
• Oracle Financials
• SAP S/4HANA
• Experience supporting AI/ML workloads
• Databricks and Cloud Certifications

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives
• Shape enterprise finance and risk data platforms
• Support regulatory, management, and analytical reporting systems

Essential Skills

• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Lakehouse Architecture
• Cloudera (CDH/CDP)
• Hive
• HBase
• Impala
• Data Warehouse & Data Lake
• Medallion Architecture
• Databricks SQL
• dbt
• AWS / Azure
• Airflow / Databricks Workflows
• CI/CD
• Terraform
• Jenkins
• Git
• Finance & Risk Data Models
• Enterprise Data Architecture








Requirements
Sailpoint
Créer une alerte emploi pour cette recherche

Principal Databricks Data Engineer • Toronto, ON, ca

Offres similaires

Principal Data Engineer (Azure)

Tiger AnalyticsToronto, ON, CA
Temps plein

Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Tiger Analytics is a global AI and analytics consulting firm.With data and technology at the core of o... Voir plus

 • Offre sponsorisée

Principal Software Engineer - Data

ZoomInfoToronto
Temps plein

We are seeking an experienced Principal Software Engineer (Data) to join our fast-paced, collaborative data team.In this role, you will have broad authority to drive the direction of our technograp... Voir plus

 • Offre sponsorisée

Junior Data Engineer

Amaris ConsultingToronto, ON, CA
Temps plein

As a Junior Data Engineer, you will join a dynamic data team and contribute to the development, optimization, and maintenance of data pipelines and data infrastructure.You will work closely with da... Voir plus

 • Offre sponsorisée

Principal Architect Focused on Cutting-Edge Data Solutions

CDW CanadaToronto, ON, CA
Temps plein

Elevate your career as a Principal Data Solutions Architect in a vibrant pre-sales capacity.Lead efforts in transforming customers' data practices through effective analytics and lakehouse strategi... Voir plus

 • Offre sponsorisée

Principal Software Engineer, BI & Data Platform

CalabrioToronto, ON, CA
Temps plein

Are you driven by innovation and looking to thrive in a fast-paced, growing environment? Join us at Calabrio‑Verint and be part of our dynamic team! Help us in reshaping the landscape of customer e... Voir plus

 • Offre sponsorisée

Data Engineer

Oxford Properties GroupToronto, ON, CA
Temps plein

Choose a workplace that empowers your impact.Join a global workplace where employees thrive.One that embraces diversity of thought, expertise and passion.A place where you can personalize your empl... Voir plus

 • Offre sponsorisée

Oliver Wyman - Principal Data Engineer - $240,000 - $260,000 A Year

Oliver WymanToronto, Canada
Temps plein

Join to apply for the Oliver Wyman - Principal Data Engineer role at Oliver Wyman Locations: Boston, New York, Chicago, Washington DC, Toronto, Montreal Who We Are Oliver Wyman Performance Tran... Voir plus

 • Offre sponsorisée

Senior Data Engineer (Databricks Focused)

BDO CanadaToronto, ON, CA
Temps plein

Putting people first, every day.BDO is a firm built on a foundation of positive relationships with our people and our clients.Each day, our professionals provide exceptional service, helping client... Voir plus

 • Offre sponsorisée

Junior Databricks Data Engineer — Hybrid Lakehouse - C$70,000 - C$94,000 A Year

The Citco GroupToronto, Canada
Temps plein

A leading financial services firm in Toronto is seeking a Junior Data Engineer to contribute to their Centralized Data Platform built on Databricks.The candidate will develop and maintain data pipe... Voir plus

 • Offre sponsorisée

Data Engineer

LeadingtalentMarkham, ON, CA
Temps plein

At IBM, work is more than a job – it’s a calling: To build.To think along with clients and sell.Not just to do something better, but to attempt things you’ve never thought possible.Are you ready to... Voir plus

 • Offre sponsorisée

Senior Data Engineer ( Databricks/Snowflake)

TEEMA Solutions GroupToronto, ON, CA
Temps plein

Toronto (3 days downtown/Hybrid).Our client is interviewing to onboard a Senior Data Engineers to join their Data & Analytics team within the Wealth Technology group.We are looking for Data Enginee... Voir plus

 • Offre sponsorisée

Principal Data Engineer

AvisoToronto, ON, CA
Temps plein

We are looking for a Principal Data Engineer to join our Data and Analytics Group within the Technology Services Partners (TSP) department.Reporting to the Sr Director, Data Platform Delivery and E... Voir plus

 • Offre sponsorisée

Data Engineer: PySpark, Databricks & Tableau

Highbrow LLCToronto, ON, CA
Temporaire

A leading data engineering firm in Toronto is seeking a Data Engineer with 2-5 years of experience in technologies such as PySpark, SQL, and Databricks.The role involves working in the Data Science... Voir plus

 • Offre sponsorisée

Data Engineer

DS Technologies IncToronto, ON, CA
Temps plein

Get AI-powered advice on this job and more exclusive features.About US: We are a company that provides innovative, transformative IT services and solutions.We are passionate about helping our clien... Voir plus

 • Offre sponsorisée

Senior Databricks Data Engineer — Lakehouse & Ai Pipelines - C$120,000 - C$150,000 A Year

The Citco Group LimitedToronto, Canada
Temps plein

Senior Data Engineer sought to design and optimize data pipelines on Databricks Lakehouse Platform. Voir plus

 • Offre sponsorisée

Lead Data Engineer – Databricks

TEEMAToronto, Ontario, Canada
Temps plein

Team Lead, Data Engineering – Databricks.Our client has provided specialist financial services to alternative investment funds, investors, multinationals, and private clients worldwide.Our data eng... Voir plus

 • Offre sponsorisée

Senior Data Engineer – Databricks & Cloud Data Platforms

ShyftLabsToronto
Temps plein

A leading data solutions provider in Toronto is seeking a skilled Data Engineer to design and optimize big data solutions using Databricks.The ideal candidate will have significant experience with ... Voir plus

 • Offre sponsorisée

Senior Data Engineer: Databricks Lakehouse & ML Pipelines

TeraWatt InfrastructureToronto, ON, CA
Temps plein

A leading infrastructure company in Toronto is seeking a Senior Data Engineer to design and implement scalable data architectures.In this role, you will work with cross-functional teams to optimize... Voir plus

 • Offre sponsorisée

Principal Data Engineer

RAVLToronto, ON, CA
Temps plein

As a Principal Data Engineering leader at RAVL, you will design, define, and elevate enterprise‑grade data and AI platforms that are secure, scalable, and high‑performance.Operating as a Community ... Voir plus

 • Offre sponsorisée

Lead Data Engineer

Thomson ReutersToronto, ON, CA
Temps plein

We are seeking a highly skilled and strategic Lead Data Engineer to join our team.In this advanced individual contributor role, you will spearhead the development of sophisticated, insightful, and ... Voir plus