Role : Lead Data Engineer - Azure Data Platform.
Location : Toronto, ON (Onsite).
Duration : Long Term Contract.
Key Responsibilities :
Data Platform Architecture :
- Architect end-to-end data pipelines using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Gen2 for enterprise-scale data processing.
- Develop production-grade PySpark transformations and Python ETL processes handling petabyte-scale datasets with complex business logic.
- Lead data modeling initiatives implementing star schemas, dimensional modeling, and SCD Type 2 for analytics and reporting platforms.
Team Leadership & Technical Excellence :
Mentor and lead data engineering teams of 5-10 engineers through code reviews, technical standards, and architectural decision-making.Implement enterprise data quality frameworks using Great Expectations, Azure Monitor, and Azure Purview for governance and compliance.Real-Time & Advanced Analytics :
Design real-time streaming solutions leveraging Azure Event Hubs, Stream Analytics, and Delta Lake for low-latency analytics use cases.Optimize Spark jobs for cost-efficiency, performance, and scalability across Azure Databricks clusters.Required Technical Expertise :
Azure Platform (5+ years) : Data Factory, Databricks, Synapse Analytics, Data Lake Gen2, Event Hubs, Stream Analytics, Purview.Python / PySpark : Pandas, NumPy, PySpark DataFrames / SQL, Delta Lake, MLflow.Data Modeling : Star Schema, Dimensional Modeling, SCD Type 2, Data Quality Frameworks.Orchestration : Airflow (MWAA), Data Factory Pipelines, Databricks Workflows.SQL & Performance : Advanced T-SQL, Spark SQL, Query Optimization.DevOps & IaC : Terraform, ARM Templates, Azure DevOps CI / CD.Monitoring : Azure Monitor, Log Analytics, Great Expectations, Monte Carlo.Leadership Requirements :
Lead developer experience managing data engineering teams through complete SDLC with demonstrated success mentoring junior engineers and driving technical excellence.Executive stakeholder communication : Translate complex technical concepts into business value during requirement sessions and roadmap presentations.Agile / Scrum mastery : Lead sprint ceremonies using JIRA, Confluence, and cross-functional collaboration with data scientists, analysts, and business teams.Experience Profile :
7+ years data engineering experience including 5+ years hands-on Azure platform expertise.Multiple long-term enterprise projects (1.5+ years each) building production data platforms.Hands-on Python and PySpark demonstrated across 3+ major projects with complex transformations.