Talent.com
Soar Consultants
Data Engineer - DataBricksSoar Consultants • Toronto, ON, CA
Data Engineer - DataBricks

Data Engineer - DataBricks

Soar Consultants • Toronto, ON, CA
30+ days ago
Job type
  • Full-time
Job description

DESCRIPTION

We are currently looking for a Data Engineer for a client in Toronto. They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sectors.

RESPONSIBILITIES

  • Analyze and understand business requirements related to data and information, with a particular focus on the financial services industry.
  • Design data integration and ETL processes using tools such as Databricks, Spark, Hadoop, and Hive, following a holistic, scalable, pragmatic, and effective approach to data processing.
  • Ingest and transform data in Databricks for batch and near real-time processing using technologies such as PySpark and SparkSQL, while leveraging orchestration tools like Azure Data Factory within the broader Microsoft Azure ecosystem. Design and implement data pipelines integrated with Continuous Integration/Continuous Delivery (CI/CD) frameworks as required (DevOps).
  • Develop automated unit-testing to ensure data quality and reliability.
  • Develop design and operational documentation for job schedules and data pipelines.
  • Produce estimates for data analysis and mapping, design, development, and testing.
  • Ensure compliance with business, data, and technical requirements.
  • Comply with client’s enterprise architecture standards, policies, and procedures are followed.

REQUIREMENTS

  • Minimum 5 years experience in Information Technology, participating in complex projects.
  • Minimum Bachelors degree in Computer Science.
  • Strong communication skills.
  • Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
  • Strong SQL knowledge of using in Databricks workspace and Delta Lake features.
  • Strong programming skills in Pyspark and Apache Spark Architecture.
  • Experience with DLT for building reliable and maintainable data pipelines for streaming data using Databricks Structured Streaming.
  • Strong understanding of at least one major cloud provider (e.g., AWS, Azure, or GCP) and its data-related services is essential. This includes knowing about services for data storage (like S3 or Azure Blob Storage) and how they integrate with Databricks.
  • Designing schemas for databases and understanding concepts like the medallion architecture (Bronze, Silver, Gold layers) and slowly changing dimensions (SCD).
  • Familiarity with concepts of data quality, security, and governance, often utilizing Databricks's Unity Catalog.
  • Knowledge of ETL development using SSIS, Informatica, Data Stage.
  • Knowledge of and experience with Copilots and LLMs to gain higher productivity.
  • Strong knowledge of Oracle, Microsoft SQL databases.
  • Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
  • Experience in .NET development and UNIX shell scripting (nice to have).
  • Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
  • Ability to work independently and excel in a team environment.
  • Preferably as Databricks Certified Data Engineer Associate.
#J-18808-Ljbffr
Create a job alert for this search

Data Engineer - DataBricks • Toronto, ON, CA

Similar jobs

Data Engineer

Xplore Inc.Markham, ON, CA
Full-time

Canada’s fibre, 5G and satellite broadband company for rural living.Xplore is committed to the relentless pursuit of an improved broadband experience for all Canadians.Xplore is building a world‑cl... Show more

 • Promoted

Data Engineer

Tata Consultancy ServicesToronto, ON, CA
Full-time

Tata Consultancy Services (TCS) is an equal opportunity employer, and embraces diversity in race, nationality, ethnicity, gender, age, physical ability, neurodiversity, and sexual orientation, to c... Show more

 • Promoted

Data Engineer

LeadingtalentMarkham, ON, CA
Full-time

At IBM, work is more than a job – it’s a calling: To build.To think along with clients and sell.Not just to do something better, but to attempt things you’ve never thought possible.Are you ready to... Show more

 • Promoted

Data Engineer

ZafinToronto, ON, CA
Full-time

Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered.Our technology enables organizations to move faster while maintaining... Show more

 • Promoted

Data Engineer

TekStaff IT SolutionsToronto, ON, CA
Full-time

TekStaff's Client has a current vacancy for "Data Engineer".Typical Day-to-Day Responsibilities: Responsible for leading technical aspects of project from inception through to deployment.Participat... Show more

 • Promoted

AWS Databrciks Engineer

AkkodisToronto, ON, CA
Full-time

Toronto, ON Hybrid (3 days Onsite- Tue, Wed, Thursday).Citizens or PR only considered for the TCS role, sorry, OWP won't work.Design, develop, and maintain scalable and robust data pipelines on Dat... Show more

 • Promoted

Senior Data Engineer ( Databricks/Snowflake)

TEEMA Solutions GroupToronto, ON, CA
Full-time

Toronto (3 days downtown/Hybrid).Our client is interviewing to onboard a Senior Data Engineers to join their Data & Analytics team within the Wealth Technology group.We are looking for Data Enginee... Show more

 • Promoted

Databricks Engineer

PeoplePilottoronto, on, Canada
Full-time

Data Warehouse and Data Lake platforms.Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.Redesign ingestion, transformation, a... Show more

 • Promoted

Data Engineer

Veeva Systems Inc.toronto, on, Canada
Full-time

We are looking for a talented and experienced data engineer to join our team.The ideal candidate will have a strong understanding of data engineering principles and practices, as well as experience... Show more

 • Promoted

Data Engineer

DS Technologies IncToronto, ON, CA
Full-time

Get AI-powered advice on this job and more exclusive features.About US: We are a company that provides innovative, transformative IT services and solutions.We are passionate about helping our clien... Show more

 • Promoted

Flinks Senior Data Engineer Role

Flinks Technology Inc.toronto, on, Canada
Full-time

Join Flinks as a Senior Data Engineer and spearhead the data and ML platform's evolution.This role is designed for those who thrive on building production-grade data solutions in a remote setup.You... Show more

 • Promoted

Data Engineer - Analytics Pipelines & Power BI (Hybrid)

Accenturetoronto, on, Canada
Full-time

A global technology consulting firm in Toronto is seeking a Data Engineer to support large-scale infrastructure programs.This role involves designing data transformation frameworks for analytics-re... Show more

 • Promoted

Data Engineer

Venn (formerly Vault)Toronto, ON, CA
Full-time

Venn is the fastest-growing B2B fintech in Canada.We're on a mission to transform business banking for Canadian entrepreneurs and are looking to hire A-players interested in disrupting traditional ... Show more

 • Promoted

Data Engineer Role in Alberta’s Digital Division

PathlionToronto, ON, CA
Full-time

Be part of Alberta's digital transformation as a Data Engineer.Contribute to innovative projects by creating data pipelines and deriving insights that meet the needs of citizens.As a core member of... Show more

 • Promoted

Data Engineer - Snowflake

Tiger AnalyticsToronto, ON, CA
Full-time

Tiger Analytics is a fast-growing advanced analytics consulting firm.Our consultants bring deep expertise in Data Science, Machine Learning and AI.We are the trusted analytics partner for several F... Show more

 • Promoted

Data Engineer

LuxoftToronto, ON, CA
Full-time

Get AI-powered advice on this job and more exclusive features.Direct message the job poster from Luxoft.You will work as a Data Engineer in Trading Technology, Public Market Investments team, partn... Show more

 • Promoted

Data Engineer

Quantum Management Services Ltd.Toronto, ON, CA
Temporary

Quantum Management Services Ltd.North York, ON (Hybrid – on-site Monday‑Thursday).We are looking for a Data Engineer to support large‑scale retail and e‑commerce data initiatives, working with comp... Show more

 • Promoted

Databricks Senior Engineer: Data-Driven Insights

WinsAboveToronto, ON, CA
Full-time

Accelerate data-driven transformation as a Senior Solutions Engineer at Databricks in Toronto, ON.Utilize your expertise in ML and AI to provide strategic guidance to large enterprises.In this infl... Show more