AI Data Engineer

AI Data Engineer

Back to All Jobs
[social_share]

Our client requires an AI Data Engineer to support a large-scale regulatory modernization initiative focused on transforming legacy systems into modern data management and geospatial platforms. The resource will design, build, and operate reliable Azure-based data pipelines and analytics-ready data products that support regulatory oversight, compliance monitoring, operational reporting, advanced analytics, and evidence-based decision-making. Success in this role requires strong experience with Microsoft Azure, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Python, SQL, data governance, data quality, APIs, and AI-assisted data engineering workflows.

Resource will work remotely but must be available to be on-site when requested for meetings. Frequency of meeting could be up 3–4 times (or more) per fiscal month. Standard Hours of work are 08:15 – 16:30 Alberta time. Work must be done from within Canada, due to network and data security issues. Resource is responsible to pay for travel to on-site meetings.

Edmonton, Alberta (Remote within Canada)

Responsibilities

• Design and implement scalable, secure, and high-performance data architecture on Microsoft Azure, supporting both cloud-native and hybrid environments.
• Lead the development of data ingestion, transformation, and integration pipelines using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
• Work with the Data Architect and manage data lakes and structured storage solutions using Azure Data Lake Storage Gen2, ensuring efficient access and governance.
• Integrate data from diverse source systems including ServiceNow, and geospatial systems, using APIs, connectors, and custom scripts.
• Develop and maintain robust data models and semantic layers to support operational reporting, analytics, and machine learning use cases.
• Build and optimize data workflows using Python and SQL for data cleansing, enrichment, and advanced analytics within Azure Databricks.
• Design and expose secure data services and APIs using Azure API Management for downstream systems.
• Implement data governance practices, including metadata management, data classification, and lineage tracking.
• Ensure compliance with privacy and regulatory standards (e.g., FOIP, GDPR) through role-based access controls, encryption, and data masking.
• Monitor and troubleshoot data pipelines and integrations
• Utilize AI and automation tools to streamline data engineering workflows, including pipeline development, testing, monitoring, and documentation.
• Leverage AI-assisted tools for code generation, optimization
• Design and curate standardized, high quality datasets that are suitable for advanced analytics and future AI use cases.

Mandatory Requirements

• Post-Secondary degree, diploma or certificate in Computer Science or related field of study.
• 1 year use of AI-Experienced in using AI for code generation, data analysis, automation, and enhancing productivity in data engineering workflows.
• 3 years’ experience building scalable data pipelines with Azure Databricks, Delta Lake, Workflows, Jobs, and Notebooks, plus cluster management. Extending solutions to Synapse Analytics and Microsoft Fabric is a plus.
• 3 years’ experience designing data solutions for analytics-ready, trusted datasets using tools like Power BI and Synapse, including semantic layers, data marts, and data products for self-service, data science, and reporting
• 2 years’ experience in data governance, security, and metadata management within a Databricks-based platform.
• 4 years’ experience in Github/Git for version control, collaborative development, code management, and integration with data engineering workflows.
• 3 years’ experience with Azure services (Storage, SQL, Synapse, networking) for scalable, secure solutions, and with authentication (Service Principals, Managed Identities) for secure access in pipelines and integrations
• 5 years’ experience in Python (including PySpark) and SQL, applied to developing, orchestrating, and optimizing enterprise-grade ETL/ELT workflows in a large-scale cloud environment.

Desirable Requirements

• 6 years’ direct, hands-on experience performing business requirement analysis related to data manipulation/transformation, cleansing and wrangling.
• 6 years’ experience and strong technical knowledge of Microsoft SQL Server, including database design, optimization, and administration in enterprise environments.
• 2 years’ experience extending or integrating data solutions with Azure Synapse Analytics and Microsoft Fabric (Lakehouse, Warehouse, Semantic Models).
• 1 year direct experience building data products in Government of Alberta cloud environment
• 2 years’ experience building scalable ETL pipelines, data quality enforcement, and cloud integration using TALEND technologies.
• 3 years skilled in building secure, scalable RESTful APIs for data exchange, with robust auth, error handling, and support for real-time automation.
• 5 years’ experience working with cross-functional teams to create software applications and data products.
• 1 year experience working with ServiceNow- Azure based Data Management Platform Integrations.
• 3 years’ experience in Message Queueing Technologies, implementing message queuing using tools like ActiveMQ and Service Bus for scalable, asynchronous communication across distributed systems.