Talent.com
Astra North Infoteck Inc.
Principal Databricks Data EngineerAstra North Infoteck Inc. • Toronto, ON, ca
Principal Databricks Data Engineer

Principal Databricks Data Engineer

Astra North Infoteck Inc. • Toronto, ON, ca
Il y a 27 jours
Type de contrat
  • Temps plein
Description de poste
Job Description

Principal Databricks Data Engineer

Experience Required: 12–18 Years

Key Requirements

• 12–18 years of overall Data Engineering experience
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
• 5+ years of hands-on experience with Databricks and Apache Spark at scale
• Strong experience modernizing legacy Cloudera platforms including:
• CDH/CDP
• Hive
• HBase
• Impala
• Spark
• Modernize Cloudera platforms to Databricks Lakehouse architecture
• Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
• Ensure data reconciliation, audit integrity, and consistency during migration
• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
• Implement layered data architecture including:
• Raw / Landing Layer
• Curated / Conformed Layer
• Semantic / Consumption Layer
• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
• Strong experience with finance and risk data models including:
• General Ledger
• Sub-ledger
• Financial Hierarchies
• Credit Risk
• Liquidity Risk
• Market Risk
• Enable reporting capabilities including:
• Aggregation
• Drill-down
• Drill-back
• Build and manage semantic and consumption layers for BI, reporting, and analytics
• Define business metrics, dimensions, hierarchies, and KPIs
• Experience with:
• Databricks SQL
• Delta Tables
• dbt or similar frameworks
• Develop and optimize large-scale data pipelines using:
• PySpark
• Spark SQL
• Delta Lake
• Implement Medallion Architecture including:
• Bronze Layer
• Silver Layer
• Gold Layer
• Optimize workloads using:
• Z-ORDER
• OPTIMIZE
• Caching
• Cluster Configuration Tuning
• Implement:
• Data Governance
• Data Quality Frameworks
• Reconciliation Controls
• Exception Handling
• Establish data lineage and metadata management
• Ensure data security, access control, and compliance standards
• Experience with AWS or Azure cloud platforms
• Experience with CI/CD pipelines using:
• Git
• Terraform
• Jenkins
• Azure DevOps
• Familiarity with:
• Apache Airflow
• Databricks Workflows
• Experience with dbt is an advantage
• Act as a technical authority and lead enterprise architecture decisions
• Mentor senior engineers and establish engineering standards
• Collaborate with finance, risk, analytics, and governance stakeholders
• Translate complex data structures into business-ready insights

Nice-to-Have Skills

• Experience in BFSI, Capital Markets, or Regulatory Reporting
• Exposure to:
• SAP Finance
• Oracle Financials
• SAP S/4HANA
• Experience supporting AI/ML workloads
• Databricks and Cloud Certifications

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives
• Shape enterprise finance and risk data platforms
• Support regulatory, management, and analytical reporting systems

Essential Skills

• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Lakehouse Architecture
• Cloudera (CDH/CDP)
• Hive
• HBase
• Impala
• Data Warehouse & Data Lake
• Medallion Architecture
• Databricks SQL
• dbt
• AWS / Azure
• Airflow / Databricks Workflows
• CI/CD
• Terraform
• Jenkins
• Git
• Finance & Risk Data Models
• Enterprise Data Architecture








Requirements
Sailpoint
Créer une alerte emploi pour cette recherche

Principal Databricks Data Engineer • Toronto, ON, ca

Offres similaires

Principal Data Engineer (Azure)

Tiger AnalyticsToronto, ON, CA
Temps plein

Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Tiger Analytics is a global AI and analytics consulting firm.With data and technology at the core of o... Voir plus

 • Offre sponsorisée

Principal Software Engineer - Data

ZoomInfoToronto
Temps plein

We are seeking an experienced Principal Software Engineer (Data) to join our fast-paced, collaborative data team.In this role, you will have broad authority to drive the direction of our technograp... Voir plus

 • Offre sponsorisée

Data Engineer: PySpark, Databricks & Tableau

Highbrow LLCToronto
Temps plein +1

A leading data engineering firm in Toronto is seeking a Data Engineer with 2-5 years of experience in technologies such as PySpark, SQL, and Databricks.The role involves working in the Data Science... Voir plus

 • Offre sponsorisée

Principal Engineer — Ai-Driven Data Extraction Platform - C$202,000 - C$278,000 A Year

Leading Data Intelligence FirmToronto County, Canada
Temps plein

Lead the design and reliability of a global financial data extraction platform, focusing on AI-driven solutions and mentoring engineers. Voir plus

 • Offre sponsorisée

Senior Databricks Data Engineer — Lakehouse & Ai Pipelines - C$120,000 - C$150,000 A Year

The Citco Group LimitedNorth York, Canada
Temps plein

Senior Data Engineer sought to design and optimize data pipelines on Databricks Lakehouse Platform. Voir plus

 • Offre sponsorisée • Nouvelle offre

Data Platform Engineer

Princeton IT Services, IncToronto, ON, CA
Temps plein

Director Data and AI, Embedded Finance.The primary focus of this role will be to design, build, and maintain the EMFI Data Application Platform, a financial analytics application.The incumbent will... Voir plus

 • Offre sponsorisée

Oliver Wyman - Principal Data Engineer - $240,000 - $260,000 A Year

Oliver WymanNorth York, Canada
Temps plein

Principal Data Engineer role at Oliver Wyman.Lead technical teams, design cloud-native solutions, and mentor staff.Experience with software engineering, Agile methodologies, and communication skill... Voir plus

 • Offre sponsorisée

Principal Data Engineer

Fitch RatingsToronto, ON, CA
Temps plein

Fitch Group is currently seeking a Associate Director/Lead Data Engineer based out of our Toronto office.As a leading, global financial information services provider, Fitch Group delivers vital cre... Voir plus

 • Offre sponsorisée

Senior Databricks Data Engineer — Lakehouse & Ai Pipelines - C$120,000 - C$150,000 A Year

Financial Services ProviderNorth York, Canada
Temps plein

Data Engineer needed in Toronto to design and optimize data pipelines on Databricks Lakehouse Platform using Python, Spark, and AWS for a financial services provider. Voir plus

 • Offre sponsorisée

Senior Data Engineer ( Databricks/Snowflake)

TEEMA Solutions GroupToronto, ON, CA
Temps plein

Toronto (3 days downtown/Hybrid).Our client is interviewing to onboard a Senior Data Engineers to join their Data & Analytics team within the Wealth Technology group.We are looking for Data Enginee... Voir plus

 • Offre sponsorisée

Principal Data Engineer

AvisoToronto, ON, CA
Temps plein

We are looking for a Principal Data Engineer to join our Data and Analytics Group within the Technology Services Partners (TSP) department.Reporting to the Sr Director, Data Platform Delivery and E... Voir plus

 • Offre sponsorisée

Data Engineer

Inizio Partners CorpToronto
Temps plein

Job Title: Consultant Data Engineer.We're looking for a Senior Data Engineer with strong data platform experience to help with support of existing AWS/dbt platform and that can help with standing u... Voir plus

 • Offre sponsorisée

Principal Engineer, Enterprise Data & Analytics - $120,000 - $150,000 A Year

QuadReal Property GroupNorth York, Canada
Temps plein

Senior technical leader responsible for data and analytics solutions, setting standards, mentoring, and hands-on delivery of complex data problems for a global real estate investment company. Voir plus

 • Offre sponsorisée

Senior AWS Databricks Data Engineer - Hybrid Toronto

AkkodisToronto, ON, CA
Temps plein

A leading IT consultancy is seeking a Mid-Senior Level Data Engineer for a contract role in Toronto.Candidates will design, develop, and maintain scalable data pipelines, ensuring data quality and ... Voir plus

 • Offre sponsorisée

Principal Data Solutions Engineer - C$120,000 - C$150,000 A Year

Leading Real Estate FirmToronto County, Canada
Temps plein

Seeking a Principal Data Solutions Engineer for a leading real estate firm to lead data analytics solutions, provide mentorship, and develop technical standards using SQL and Python. Voir plus

 • Offre sponsorisée

Principal Data Engineer - Tangerine

Scotiabanktoronto, on, Canada
Temps plein

Tangerine is Canada’s leading direct bank.We offer flexible and accessible banking options, innovative products, and award-winning client service.Our employees come to work each day to help Canadia... Voir plus

 • Offre sponsorisée

Data Engineer

Adastra Corporationmarkham, on, Canada
Temps plein

Adastra is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms that support advanced analytics, reporting, and AI initiatives.This r... Voir plus

 • Offre sponsorisée

Databricks Engineer

PeoplePilotToronto, Ontario, Canada
Temps plein

Data Warehouse and Data Lake platforms.Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.Redesign ingestion, transformation, a... Voir plus

 • Offre sponsorisée

Affinity Principal Engineer Data Solutions

AffinityToronto, Canada
Temps plein

Location : Preference to have candidates based in Toronto, next best, candidates would be located in Vancouver.Hybrid work environment of 1 - 2 days onsite.On behalf of our client, Affinity is seek... Voir plus

 • Offre sponsorisée

Principal Engineer — Ai-Driven Data Extraction Platform - C$202,000 - C$278,000 A Year

Data Intelligence FirmNorth York, Canada
Temps plein

Lead the design and reliability of a global financial data extraction platform, focusing on AI-driven solutions and system architecture for a leading data intelligence firm. Voir plus