Talent.com
Acestack
Data Engineer – Kafka / PySpark / HadoopAcestack • Toronto, ON, Canada
Data Engineer – Kafka / PySpark / Hadoop

Data Engineer – Kafka / PySpark / Hadoop

Acestack • Toronto, ON, Canada
Il y a 2 jours
Type de contrat
  • Temps plein
  • Quick Apply
Description de poste
  • Job Title: Data Engineer Kafka / PySpark / Hadoop
  • Location: Toronto, ON
  • Work Model: Onsite
  • Job Type: Full Time (FTE)

Job Description

We are seeking an experienced Data Engineer with strong hands-on expertise in Kafka, PySpark, Python, and Hadoop to design, develop, and support scalable batch and real-time data pipelines. The ideal candidate will have strong experience working with large-scale distributed data processing environments and enterprise data integration solutions.

Key Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines.
  • Develop data processing applications using Python and PySpark/Apache Spark.
  • Build and support Kafka-based data ingestion and streaming pipelines.
  • Work with Hadoop and related technologies to process large volumes of data.
  • Develop and maintain ETL/ELT pipelines for data ingestion, transformation, cleansing, and integration.
  • Perform data validation, reconciliation, and quality checks.
  • Troubleshoot pipeline failures, data discrepancies, and performance issues.
  • Optimize Spark/PySpark jobs and SQL queries for performance and scalability.
  • Monitor data pipelines and resolve production issues.
  • Collaborate with data architects, developers, analysts, and business teams.
  • Participate in Agile development, testing, deployment, and production support activities.

Required Skills

  • Strong hands-on experience with Python for data engineering and automation.
  • Strong expertise in PySpark / Apache Spark.
  • Hands-on experience with Apache Kafka for real-time data ingestion and streaming.
  • Strong experience with the Hadoop ecosystem and distributed data processing.
  • Strong SQL skills and experience working with large datasets.
  • Experience developing and maintaining ETL/ELT data pipelines.
  • Strong understanding of distributed computing and data processing concepts.
  • Experience with data ingestion, transformation, cleansing, and integration.
  • Strong troubleshooting and performance optimization skills.

Good to Have

  • Hive
  • Databricks
  • AWS, Azure, or GCP
  • Git and CI/CD
  • Unix/Linux
  • Airflow or Autosys
  • Relational and NoSQL databases
Créer une alerte emploi pour cette recherche

Data Engineer – Kafka / PySpark / Hadoop • Toronto, ON, Canada

Offres similaires

Databricks Data Engineer - ETL Lead (Hybrid Toronto)

SPECTRAFORCEToronto
Temps plein

A leading financial technology provider in Canada is seeking a Mid-Senior level professional to lead ETL development and manage a Databricks platform for Capital Markets.This role requires strong e... Voir plus

 • Offre sponsorisée

Data Engineer: Cloud Pipelines & BigQuery

Quantum Management Services Ltd.Toronto
Temps plein +1

A data management firm located in Ontario is looking for a Data Engineer to support retail and e-commerce data initiatives.This position involves designing and maintaining scalable data pipelines, ... Voir plus

 • Offre sponsorisée

Senior AWS Databricks Data Engineer - Hybrid Toronto

AkkodisToronto
Temps plein

A leading IT consultancy is seeking a Mid-Senior Level Data Engineer for a contract role in Toronto.Candidates will design, develop, and maintain scalable data pipelines, ensuring data quality and ... Voir plus

 • Offre sponsorisée

Lead Data Engineer Role at Paytm Labs

PaytmToronto
Temps plein

Become a key player at Paytm Labs as a Lead Data Engineer.Drive innovations in data engineering through designing pipelines that handle petabytes of data efficiently in a hybrid work environment.As... Voir plus

 • Offre sponsorisée

Senior Data Engineer (Python, Spark, Snowflake) - Toronto, ON

TechedinToronto, ON, CA
Temps plein

Job Title: Senior Data Engineer (Python, Spark, Snowflake) – Location: Hybrid (2 days remote), Toronto, Canada – Experience: 5+ Years – Rate: 65-72 CAD per hour.We are hiring for the Senior Data En... Voir plus

 • Offre sponsorisée

Senior Data Platform Engineer: Cloud Data & Analytics

GravityToronto
Temps plein

A data solutions company in Toronto is looking for a Senior Data Platform Engineer to design, build, and scale data platforms for mission-critical government technology solutions.The ideal candidat... Voir plus

 • Offre sponsorisée

Senior Data Engineer

NEARSOURCE TECHNOLOGIESToronto, ON, CA
Temps plein

Job Title: Senior Data Engineer.Job Location: Hybrid, Toronto, Ontario, Canada.NearSource is looking for a Data Platform Engineer with expertise in Python, Spark, and Snowflake to join our engineer... Voir plus

 • Offre sponsorisée

Senior Data Engineer: Snowflake, DBT & Glue Pro (Contract)

Compunnel Inc.Markham, York Region, CA
Temps plein

A leading IT Services and Consulting firm is seeking a Senior Data Engineer in Markham, Ontario.Candidates should have over 8 years of data engineering experience and proficiency in technologies su... Voir plus

 • Offre sponsorisée

Data Engineer - DataBricks

Soar ConsultantsToronto, Ontario, Canada
Temps plein

We are currently looking for a Data Engineer for a client in Toronto.They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sector... Voir plus

 • Offre sponsorisée

Senior Data Engineer: Pyspark, Delta Lake & Azure Lakehouse - C$95,000 - C$145,000 A Year

Global Technology Service ProviderToronto, Canada
Temps plein

A global technology service provider in Toronto is seeking a Data Engineer to lead the development of ETL/ELT pipelines and ensure high standards in code quality.The ideal candidate has deep expert... Voir plus

 • Offre sponsorisée

Senior Data Engineer - Snowflake, AWS & Spark

Tiger AnalyticsToronto, ON, CA
Temps plein

An analytics consulting firm in Toronto is seeking a Data Engineer to architect and implement advanced analytics capabilities.The ideal candidate will have over 12 years of experience in data engin... Voir plus

 • Offre sponsorisée

Data Engineer & Analytics

Kredivo GroupMarkham, York Region, CA
Temps plein

As a Data Engineer and Analytic you will be tasked with empowering internal stakeholders to delve deep into the complexities of the data in our warehouse.You will work closely with Product, Enginee... Voir plus

 • Offre sponsorisée

Senior Cloud Data Engineer

Rubicon PathToronto, Ontario, Canada
Temps plein

Become a pivotal Senior Software Developer focused on data engineering.Leverage your strong background in Python and AWS to create innovative data solutions in a hybrid setup.In this role, you will... Voir plus

 • Offre sponsorisée

Senior Snowflake Data Engineer | Aws, Spark & Ai Growth

Tiger Analytics, LLCToronto, Canada
Temps plein

A fast-growing analytics consulting firm is seeking a Data Engineer to architect and implement advanced analytics capabilities.The role requires expertise in data engineering with extensive experie... Voir plus

 • Offre sponsorisée

Senior Engineer in Data Solutions

Canadian Tire CorporationToronto, Ontario, Canada
Temps plein

Drive innovation in data solutions at Canadian Tire Corporation as a Senior Data Engineer in Toronto, ON.Your role will focus on developing automated data processes for enhanced analytics.As part o... Voir plus

 • Offre sponsorisée

Senior Databricks Data Engineer — Lakehouse & Ai Pipelines - C$120,000 - C$150,000 A Year

Financial Services ProviderToronto, Canada
Temps plein

A leading financial services provider in Toronto seeks a Senior Data Engineer to advance its data initiatives.This role involves designing and optimizing data pipelines on the Databricks Lakehouse ... Voir plus

 • Offre sponsorisée

Remote Lead Data Engineer - Databricks & PySpark

LumenaltaToronto, ON, CA
Télétravail
Temps plein

A technology company is seeking an experienced Data Engineering Lead to guide technical direction for critical initiatives in data engineering.This fully remote position requires 10+ years of exper... Voir plus

 • Offre sponsorisée

Databricks Senior Engineer: Data-Driven Insights

WinsAboveToronto, ON, CA
Temps plein

Accelerate data-driven transformation as a Senior Solutions Engineer at Databricks in Toronto, ON.Utilize your expertise in ML and AI to provide strategic guidance to large enterprises.In this infl... Voir plus

 • Offre sponsorisée

Data Engineer – Scalable Pipelines & Bi Dashboards - $88,000 - $128,700 A Year

A technology companyToronto, Canada
Temps plein

A technology company is looking for a Data Engineer with expertise in big data systems and data processing.The role involves building resilient data infrastructure and creating dashboards for data ... Voir plus

 • Offre sponsorisée

Senior Data Engineer Focusing on Data Analytics and Collaborative Solutions

Avison YoungToronto, Ontario, Canada
Temps plein

Become a Senior Data Engineer and work at the nexus of data analytics and business collaboration.Utilize your Snowflake and dbt skills in a fast-paced, remote setting to drive project outcomes.This... Voir plus