Data Engineer

Job Title Azure Data Engineer

Job Location: Toronto, ON

Job Type: Full Time

Experience: 7+ Years

Skills Required:

• Strong understanding of data warehousing, data lakes, and best practices for data quality, security, and performance.

• Design and Development Design, develop, deploy, and maintain scalable data pipelines and ETL

• ELT processes using Azure Data Factory (ADF), Azure Databricks, and other Azure data services (e.g., Azure Data Lake Storage, Azure Synapse Analytics).

• Python, PySpark Programming Implement efficient, reusable, and testable code for data transformation, cleansing, and analysis using Python and PySpark within Databricks notebooks and jobs.

• Orchestration and Integration Use ADF pipelines to orchestrate data movement and trigger Databricks notebooks or jobs, ensuring seamless integration between various data sources and destinations.

• Data Management Manage data storage, ensure data integrity, optimize data processing, and troubleshoot performance issues related to Databricks and ADF solutions.

• Collaboration Collaborate closely with data scientists, data analysts, and business stakeholders to gather requirements and translate business needs into technical specifications and data solutions. Monitoring Security Implement monitoring, error handling, and security best practices across all data workflows and ensure compliance with data governance standards.

• Documentation Create and maintain comprehensive technical documentation for data architectures, pipelines, and processes.

• Required Skills and Qualifications Experience Proven experience in data engineering, data warehousing, or similar