Data Engineer Databricks, Python

Job Title

Data Engineer Databricks & Python

Location

Mississauga

Experience

46 years of professional experience in data engineering or data management

Key Responsibilities

Design build and maintain data pipelines using Databricks PySpark and Python

Support data warehousing initiatives including data integration transformation and cleansing

Develop and maintain data quality rules validations and mappings to ensure data accuracy and consistency

Write and optimize complex SQL queries for data extraction transformation and analysis

Troubleshoot and resolve data-related production issues ensuring system stability and data integrity

Work with cloud-based data platforms to support scalable data processing solutions

Collaborate with cross-functional teams to follow software development lifecycle (SDLC) best practices

Participate in Agile / Scrum ceremonies including sprint planning stand-ups and retrospectives

Support CI / CD pipelines for data workloads and automation

Provide ongoing operational support and continuous improvement of data systems

Required Skills & Qualifications

46 years of experience in data engineering data management or software development

Strong experience with Databricks Python and PySpark

Proficiency in writing complex SQL

Experience with data warehousing data integration and data cleansing

Working knowledge of cloud technologies

Experience troubleshooting data issues in Oracle and MS SQL environments (preferred)

Familiarity with CI / CD pipelines and automation for data workflows

Understanding of SDLC and Agile / Scrum methodologies

Strong problem-solving skills and ability to work effectively in a team environment

Nice-to-Have Skills

Experience with Perl

Exposure to DevOps practices for data platforms

Experience supporting production environments

Key Skills

Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala

Employment Type : Full Time

Experience : years

Vacancy : 1