DataOps/Cloud Data Engineer – Senior (Databricks/Oracle/Pyspark)

Job Title: DataOps/Cloud Data Engineer Senior (Databricks/Oracle/Pyspark)

Location: Toronto, ON

Work Model: Hybrid

Duration: 6 months

Extension: 3 Times

Must Haves

• Design, build, automate and optimize complex data ETL/ELT processes using Databricks, Delta Live Tables (DLT) and equivalent.
• Development of unified data platforms using Databricks and Unity catalog or its equivalent.
• Relational databases (Oracle, MySQL, SQL Server), data modelling (relational & dimensional), advanced SQL, query optimization, data replication, administration.
• Advanced SQL skills (PL/SQL, TSQL).
• Big Data processing Frameworks (PySpark).
• Proficient in creating Azure Data Factory pipelines for copy activity and custom data pipelines in Databricks with CI CD (Azure DevOPS Git Integration) .
• Data migration and data integration across different platforms Oracle to Azure Data Lake, Oracle to Databricks, Oracle to Microsoft Fabric.

Description / Responsibilities / Skills

Description

• To respond to the priorities of the Government of Ontario through the passage of the Bill 5, Protect Ontario by Unleashing Our Economy Act 2025, the resource may be partially assigned to Bill 5 related projects or initiatives on an as needed basis.
• DataOps developer that will develop programs to carry out data ingestion from existing MECP applications to Azure Data lakehouse.

Responsibilities

• Designing and developing data pipelines from source to end user
• Optimizing data pipelines
• Experience in DataOPS principle, best practice, and implementation and Agile project development and deployment
• Experience in Continuous Integration/Continuous Development/Deployment (CI/CD) and Data provisioning Automation
• Experience managing Cloud Data services for project delivery, including storage, repositories, Data Lake, Data Lakehouse, key vault, virtual machine, disk, etc.
• Experience with DataOPS performance monitoring and tuning

General Skills

• Experience with Cloud data platforms, data management and data exchange tools and technologies
• Experience with digital product, data analysis, data exchange, data provisioning, and data security
• Experience in design, development and implementation of fact/dimension model, data mapping, data warehouse, data lake and data lakehouse for enterprise
• Experience with structured, semi-structured, unstructured data collection, ingestion, provisioning and exchange technological development of enterprise data warehouse and data lake and data lakehouse solutions and operational support.
• A team player with a track record for meeting deadlines

Experience & Skill Set Requirements

Desirable Skills and Experience 25%

• Data lake design and development with focus on extracting, transforming data coming from various data sources and loading into Lakehouse in medallion architecture.
• Cloud technologies (Azure, Google, AWS).
• PL/SQL for data extraction, transformation, and loading, initial setup and other ETL experience management and support, such as troubleshooting, performance tuning, failover and recovery.
• Scripting languages (Python, Unix shell, Scala).