Senior Data Engineer

CPUS Engineering Staffing Solutions Inc.
700 University Ave (100% Remote)
$110K a year (estimated)
Remote
Full-time

Job Overview

  • Design, build and productionize modular and scalable data pipelines and data infrastructure leveraging the wide range of data sources across the organization.
  • Implement curated common data models that offer an integrated, business-centric single source of truth for business intelligence, analytics, artificial intelligence, and other downstream system use.
  • Identify, design, and implement internal process improvements : automating manual processes, optimizing data delivery, re designing infrastructure for greater scalability, etc.
  • Work with tools in the Microsoft Stack; Azure Data Factory, Azure Data Lake, Azure SOL Databases, Azure Data Warehouse, Azure Synapse Analytics Services, Azure Databricks, Microsoft Purview, and Power Bl
  • Work within an agile work management framework in delivery of products and services, including contributing to feature & user story backlog item development.
  • Develop optimized, performant data pipelines and models at scale using technologies such as Python, Spark and SOL, consuming data sources in XML, CSV, JSON, REST APls, or other formats.
  • Implement orchestration of data pipeline execution to ensure data products meet customer latency expectations, dependencies are managed, and datasets are as up to date as possible, with minimal disruption to end-customer use.
  • Create tooling to help with day-to-day tasks and reduce toil via automation wherever possible.
  • Build continuous integration / continuous delivery (Cl / CD) pipelines to automate testing and deployment of infrastructure / code.
  • Monitor the ongoing operation of in-production solutions, assist in troubleshooting issues, and provide Tier 2 support for datasets produced by the team, on an as-required basis.
  • Write and perform automated unit and regression testing for data product builds, assist with user acceptance testing and system integration testing as required, and assist in design of relevant test cases.
  • Participate in code review as both a submitter and reviewer.

Qualifications

  • 4 years of University in computer science, computer / software engineering or other relevant programs within data engineering, data analysis, artificial intelligence, or machine learning.
  • Requires experience of over 4 years and up to 6 years in data modeling, data warehouse design and data solution architecture in a Big Data environment.
  • Experience as a Data Engineer in a Big Data environment.
  • Experience with integrating structured and unstructured data across various platforms and sources.
  • Knowledge of content fragmentation, partitioning, query parallelism, and query execution plans.
  • Experience with implementing event-driven (pub / sub), near-real-time, or streaming data solutions.
  • Strong knowledge of programming methodologies (source / version control, continuous integration / continuous delivery, automated testing, quality assurance) and agile development methodologies.
  • Fluency with SQL, Python, and Spark / PySpark is required.
  • Experience with Airflow and DBT.
  • A period of over 4 years and up to and including 6 years in data ingestion, data modelling, data engineering, and software development.
  • 30+ days ago
Related jobs
Scotiabank
Toronto, Ontario

GTT is searching for Senior Data Engineer who are continuous learners and are eager to develop & boost capabilities of our existing and growing google cloud platforms. Strong experience in GCP cloud-based data engineering development (Pub/Sub, Big Query, GCS, Dataflow, Cloud Composer, DBT etc. S...

Lorven Technologies
Toronto, Ontario

Senior Risk Data Engineer / Risk Management Data Analyst . Strong technical skills (working with SQL, data handling, data mapping). Strong data analysis skills are required including analyzing large data sets in using SQL queries and other modeling tools. Producing and using Data Catalogues, underst...

theScore
Toronto, Ontario

As part of the theScore Data Engineering team, you will be working with a team of smart, friendly, and dedicated Data Engineers, Analysts, and Data Scientists determined to develop high-quality and sustainable data-driven solutions to improve profitability, growth and the user experience. The ideal ...

StackAdapt
Canada

We're looking to add Senior and Staff Data Engineers to our data science team! This team works on solving complex problems for StackAdapt's digital advertising platform. You'll be working directly with our data scientists, data engineers, Engineering team, and CTO on building pipelines and ad optimi...

RBC - Royal Bank
Toronto, Ontario

Senior Machine Learning and Data Engineer is responsible for developing application for large-scale data processing and analysis. E2E technical competency including conducting data analysis, data preprocessing, and feature engineering to prepare datasets for model training. Work alongside data scien...

Astra North Infoteck Inc.
Toronto, Ontario

Kubernetes Clusters, Cloud, Data Analytics, CEPH. Kubernetes, Docker, Ansible, HELM, Bitbucket,Jfrog, Airflow, CI/CD, Data and Analytics tech, Grafana. ...

Aritzia
Vaughan, Ontario

Proven skills, education, and/or applicable certifications in: Experience creating custom data visualizations and data experiences using modern tools (Tableau, Looker, Power BI)Ability to lead the design process from ideation to polished design, using sketches, wireframes, UX scenarios and mockups t...

Moneris
Toronto, Ontario

Do you have experience building data pipelines and transforming raw data into relevant insights? Are you an expert in Snowflake development, with an understanding of cloud platforms and big data tools? If so, we want to hear from you!. We are looking for a Senior Data Engineer to join our growing te...

Faire
Canada

Our experienced data scientists and machine learning engineers are developing solutions related to discovery, ranking, search, recommendations, ads, logistics, underwriting, and more - all with the goal of helping local retail thrive. Our team already includes experienced Data Scientists and Machine...

Zonda
Canada Remote
Remote

As a Senior Data Engineer, you will play a crucial role in designing, implementing, and optimizing our data infrastructure. Zonda is looking for a passionate Senior Data Engineer to evolve and expand our team. Work closely with data engineering to design and implement data models that align with bus...