Senior Data Engineer

Akkodis is seeking a Data Engineer for a contract position with a client in GTA and is ideally looking for experienced Azure data bricks, Hadoop, Spark, Kafka

Position : Data Engineer

Location : GTA

Contract

What You’ll Do :

• Build, Orchestrate and optimize data pipelines to extract, transform, and load (ETL) data from various sources into the organization's data warehouse or data lake.
• Integrate data from diverse sources such as databases, APIs, streaming platforms, and file systems into cohesive data pipelines
• Implement data integration solutions that support real-time, batch, and incremental data processing.
• Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data.
• Develop and maintain high-performance data pipelines that seamlessly integrate data from various sources into data warehouses, data lakes, and other storage solutions.
• Develop monitoring and alerting mechanisms to identify and address data quality issues proactively.
• Optimize ETL (Extract, Transform, Load) processes for efficiency, reliability, and data quality.
• Implement and manage data storage solutions, including relational databases, NoSQL databases, and distributed file systems.
• Manage the infrastructure and resources required to support data engineering workflows, including compute clusters, storage systems, and data processing frameworks
• Implement security controls and data governance measures for an application to protect sensitive data and ensure compliance with regulatory requirements such as GDPR, CCPA, HIPAA, and PCI-DSS. Implement encryption, access controls, and auditing mechanisms to safeguard data privacy and integrity.
• Write production-ready, testable code that adheres to engineering best practices and accounts for edge cases and error handling.
• Ensure high-quality deliverables through code reviews, design reviews, and adherence to architectural guidelines.
• Develop comprehensive unit tests and integration tests to validate data pipeline functionality and data integrity.
• Stay up to date with the latest data engineering tools, technologies, and methodologies, and evaluate their applicability to the team's needs.
• Provide technical guidance and mentorship to junior engineers on the team.
• Collaborate with data scientists, business analysts, and other stakeholders to understand data requirements and translate them into robust engineering solutions.
• Work closely with other engineering teams to integrate data solutions seamlessly into the overall technology ecosystem.
• Participate actively in agile ceremonies, communicate progress, and manage dependencies effectively.

What You’ll Bring :

Additional Skills :