Talent.com
Acestack
Data Engineer – Kafka / PySpark / HadoopAcestack • Toronto, ON, Canada
Data Engineer – Kafka / PySpark / Hadoop

Data Engineer – Kafka / PySpark / Hadoop

Acestack • Toronto, ON, Canada
2 days ago
Job type
  • Full-time
  • Quick Apply
Job description
  • Job Title: Data Engineer Kafka / PySpark / Hadoop
  • Location: Toronto, ON
  • Work Model: Onsite
  • Job Type: Full Time (FTE)

Job Description

We are seeking an experienced Data Engineer with strong hands-on expertise in Kafka, PySpark, Python, and Hadoop to design, develop, and support scalable batch and real-time data pipelines. The ideal candidate will have strong experience working with large-scale distributed data processing environments and enterprise data integration solutions.

Key Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines.
  • Develop data processing applications using Python and PySpark/Apache Spark.
  • Build and support Kafka-based data ingestion and streaming pipelines.
  • Work with Hadoop and related technologies to process large volumes of data.
  • Develop and maintain ETL/ELT pipelines for data ingestion, transformation, cleansing, and integration.
  • Perform data validation, reconciliation, and quality checks.
  • Troubleshoot pipeline failures, data discrepancies, and performance issues.
  • Optimize Spark/PySpark jobs and SQL queries for performance and scalability.
  • Monitor data pipelines and resolve production issues.
  • Collaborate with data architects, developers, analysts, and business teams.
  • Participate in Agile development, testing, deployment, and production support activities.

Required Skills

  • Strong hands-on experience with Python for data engineering and automation.
  • Strong expertise in PySpark / Apache Spark.
  • Hands-on experience with Apache Kafka for real-time data ingestion and streaming.
  • Strong experience with the Hadoop ecosystem and distributed data processing.
  • Strong SQL skills and experience working with large datasets.
  • Experience developing and maintaining ETL/ELT data pipelines.
  • Strong understanding of distributed computing and data processing concepts.
  • Experience with data ingestion, transformation, cleansing, and integration.
  • Strong troubleshooting and performance optimization skills.

Good to Have

  • Hive
  • Databricks
  • AWS, Azure, or GCP
  • Git and CI/CD
  • Unix/Linux
  • Airflow or Autosys
  • Relational and NoSQL databases
Create a job alert for this search

Data Engineer – Kafka / PySpark / Hadoop • Toronto, ON, Canada

Similar jobs

Senior Engineer in Data Solutions

Canadian Tire CorporationToronto, ON, CA
Full-time

Drive innovation in data solutions at Canadian Tire Corporation as a Senior Data Engineer in Toronto, ON.Your role will focus on developing automated data processes for enhanced analytics.As part o... Show more

 • Promoted

Data Engineer: Cloud Pipelines & BigQuery

Quantum Management Services Ltd.Toronto, ON, CA
Full-time +1

A data management firm located in Ontario is looking for a Data Engineer to support retail and e-commerce data initiatives.This position involves designing and maintaining scalable data pipelines, ... Show more

 • Promoted

Senior Data Engineer

MorningstarToronto, ON, CA
Full-time

The ideal candidate should have strong hands‑on expertise in.Design, build, and maintain end‑to‑end data pipelines with Snowflake as the primary data warehouse.Implement and support Medallion Archi... Show more

 • Promoted

Advanced Data Engineer for Large Datasets and ETL Pipelines

LumenaltaToronto, Ontario, Canada
Full-time

Transform data into insights as an Advanced Data Engineer specializing in complex ETL pipelines.Embrace this remote opportunity to use your Python or Java and SQL skills effectively.This role invit... Show more

 • Promoted

Senior Data Engineer (Python, Spark, Snowflake) - Toronto, ON

TechedinToronto, ON, CA
Full-time

Job Title: Senior Data Engineer (Python, Spark, Snowflake) – Location: Hybrid (2 days remote), Toronto, Canada – Experience: 5+ Years – Rate: 65-72 CAD per hour.We are hiring for the Senior Data En... Show more

 • Promoted

Senior Data Engineer Focusing on Data Analytics and Collaborative Solutions

Avison YoungToronto, ON, CA
Full-time

Become a Senior Data Engineer and work at the nexus of data analytics and business collaboration.Utilize your Snowflake and dbt skills in a fast-paced, remote setting to drive project outcomes.This... Show more

 • Promoted

Senior Data Engineer

NEARSOURCE TECHNOLOGIESToronto, ON, CA
Full-time

Job Title: Senior Data Engineer.Job Location: Hybrid, Toronto, Ontario, Canada.NearSource is looking for a Data Platform Engineer with expertise in Python, Spark, and Snowflake to join our engineer... Show more

 • Promoted

Senior Data Engineer: Snowflake, DBT & Glue Pro (Contract)

Compunnel Inc.Markham, York Region, CA
Full-time

A leading IT Services and Consulting firm is seeking a Senior Data Engineer in Markham, Ontario.Candidates should have over 8 years of data engineering experience and proficiency in technologies su... Show more

 • Promoted

Data Engineer - DataBricks

Soar ConsultantsToronto, Ontario, Canada
Full-time

We are currently looking for a Data Engineer for a client in Toronto.They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sector... Show more

 • Promoted

Senior AWS Databricks Data Engineer - Hybrid Toronto

AkkodisToronto, Ontario, Canada
Full-time

A leading IT consultancy is seeking a Mid-Senior Level Data Engineer for a contract role in Toronto.Candidates will design, develop, and maintain scalable data pipelines, ensuring data quality and ... Show more

 • Promoted

Senior Data Engineer - Snowflake, AWS & Spark

Tiger AnalyticsToronto, ON, CA
Full-time

An analytics consulting firm in Toronto is seeking a Data Engineer to architect and implement advanced analytics capabilities.The ideal candidate will have over 12 years of experience in data engin... Show more

 • Promoted

Data Engineer & Analytics

Kredivo GroupMarkham, York Region, CA
Full-time

As a Data Engineer and Analytic you will be tasked with empowering internal stakeholders to delve deep into the complexities of the data in our warehouse.You will work closely with Product, Enginee... Show more

 • Promoted

Senior Cloud Data Engineer

Rubicon PathToronto, Ontario, Canada
Full-time

Become a pivotal Senior Software Developer focused on data engineering.Leverage your strong background in Python and AWS to create innovative data solutions in a hybrid setup.In this role, you will... Show more

 • Promoted

Senior Data Platform Engineer: Cloud Data & Analytics

GravityToronto, ON, CA
Full-time

A data solutions company in Toronto is looking for a Senior Data Platform Engineer to design, build, and scale data platforms for mission-critical government technology solutions.The ideal candidat... Show more

 • Promoted

Senior Data Engineer — Lead High-Performance Data Pipelines

Aarorn Technologies IncMarkham, York Region, CA
Full-time

A leading tech company is seeking a Senior Data Engineer to lead data pipeline development and ensure high-quality data delivery.The ideal candidate should have over 8 years of experience in data e... Show more

 • Promoted

Lead Data Engineer Role at Paytm Labs

PaytmToronto, ON, CA
Full-time

Become a key player at Paytm Labs as a Lead Data Engineer.Drive innovations in data engineering through designing pipelines that handle petabytes of data efficiently in a hybrid work environment.As... Show more

 • Promoted

Databricks Data Engineer - ETL Lead (Hybrid Toronto)

SPECTRAFORCEToronto, Ontario, Canada
Full-time

A leading financial technology provider in Canada is seeking a Mid-Senior level professional to lead ETL development and manage a Databricks platform for Capital Markets.This role requires strong e... Show more

 • Promoted

Senior Snowflake Data Engineer | AWS, Spark & AI Growth

Tiger Analytics, LLCToronto, Ontario, Canada
Full-time

A fast-growing analytics consulting firm is seeking a Data Engineer to architect and implement advanced analytics capabilities.The role requires expertise in data engineering with extensive experie... Show more

 • Promoted

Remote Lead Data Engineer - Databricks & PySpark

LumenaltaToronto, ON, CA
Remote
Full-time

A technology company is seeking an experienced Data Engineering Lead to guide technical direction for critical initiatives in data engineering.This fully remote position requires 10+ years of exper... Show more

 • Promoted

Databricks Senior Engineer: Data-Driven Insights

WinsAboveToronto, ON, CA
Full-time

Accelerate data-driven transformation as a Senior Solutions Engineer at Databricks in Toronto, ON.Utilize your expertise in ML and AI to provide strategic guidance to large enterprises.In this infl... Show more