Talent.com
Soar Consultants
Data Engineer - DataBricksSoar Consultants • Toronto, Ontario, CA
No longer accepting applications
Data Engineer - DataBricks

Data Engineer - DataBricks

Soar Consultants • Toronto, Ontario, CA
30+ days ago
Job type
  • Full-time
Job description

DESCRIPTION

We are currently looking for a Data Engineer for a client in Toronto. They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sectors.

RESPONSIBILITIES

  • Analyze and understand business requirements related to data and information, with a particular focus on the financial services industry.
  • Design data integration and ETL processes using tools such as Databricks, Spark, Hadoop, and Hive, following a holistic, scalable, pragmatic, and effective approach to data processing.
  • Ingest and transform data in Databricks for batch and near real-time processing using technologies such as PySpark and SparkSQL, while leveraging orchestration tools like Azure Data Factory within the broader Microsoft Azure ecosystem. Design and implement data pipelines integrated with Continuous Integration/Continuous Delivery (CI/CD) frameworks as required (DevOps).
  • Develop automated unit-testing to ensure data quality and reliability.
  • Develop design and operational documentation for job schedules and data pipelines.
  • Produce estimates for data analysis and mapping, design, development, and testing.
  • Ensure compliance with business, data, and technical requirements.
  • Comply with client’s enterprise architecture standards, policies, and procedures are followed.

REQUIREMENTS

  • Minimum 5 years experience in Information Technology, participating in complex projects.
  • Minimum Bachelors degree in Computer Science.
  • Strong communication skills.
  • Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
  • Strong SQL knowledge of using in Databricks workspace and Delta Lake features.
  • Strong programming skills in Pyspark and Apache Spark Architecture.
  • Experience with DLT for building reliable and maintainable data pipelines for streaming data using Databricks Structured Streaming.
  • Strong understanding of at least one major cloud provider (e.g., AWS, Azure, or GCP) and its data-related services is essential. This includes knowing about services for data storage (like S3 or Azure Blob Storage) and how they integrate with Databricks.
  • Designing schemas for databases and understanding concepts like the medallion architecture (Bronze, Silver, Gold layers) and slowly changing dimensions (SCD).
  • Familiarity with concepts of data quality, security, and governance, often utilizing Databricks's Unity Catalog.
  • Knowledge of ETL development using SSIS, Informatica, Data Stage.
  • Knowledge of and experience with Copilots and LLMs to gain higher productivity.
  • Strong knowledge of Oracle, Microsoft SQL databases.
  • Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
  • Experience in .NET development and UNIX shell scripting (nice to have).
  • Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
  • Ability to work independently and excel in a team environment.
  • Preferably as Databricks Certified Data Engineer Associate.

#J-18808-Ljbffr

Create a job alert for this search

Data Engineer - DataBricks • Toronto, Ontario, CA

Similar jobs

Data Engineer

Tata Consultancy ServicesToronto, ON, CA
Full-time

Tata Consultancy Services (TCS) is an equal opportunity employer, and embraces diversity in race, nationality, ethnicity, gender, age, physical ability, neurodiversity, and sexual orientation, to c... Show more

 • Promoted

Data Engineer I

TDToronto, ON, CA
Full-time

Work Location:Toronto, Ontario, CanadaHours:37.Line of Business:Technology SolutionsPay Details:$69,700 - $98,400 CADTD is committed to providing fair and equitable compensation opportunities to al... Show more

 • Promoted

Data Engineer

LeadingtalentMarkham, ON, CA
Full-time

At IBM, work is more than a job – it’s a calling: To build.To think along with clients and sell.Not just to do something better, but to attempt things you’ve never thought possible.Are you ready to... Show more

 • Promoted

Data Engineer

TekStaff IT SolutionsToronto, ON, CA
Full-time

TekStaff's Client has a current vacancy for "Data Engineer".Typical Day-to-Day Responsibilities: Responsible for leading technical aspects of project from inception through to deployment.Participat... Show more

 • Promoted

AWS Databrciks Engineer

AkkodisToronto, ON, CA
Full-time

Toronto, ON Hybrid (3 days Onsite- Tue, Wed, Thursday).Citizens or PR only considered for the TCS role, sorry, OWP won't work.Design, develop, and maintain scalable and robust data pipelines on Dat... Show more

 • Promoted

Lead Data Engineer

RBCToronto, ON, CA
Full-time

Are you a hands-on data platform engineer who thrives on building cloud-native, high-scale data platforms and enabling teams on top of them? Come join us!.Are you a hands-on data platform engineer ... Show more

 • Promoted

Data Engineer

Peak Power IncToronto, ON, CA
Full-time

Peak Power is a North American cleantech company providing end-to-end battery energy storage solutions for large commercial, industrial, and manufacturing facilities operating in IESO, ISO-NE, PJM,... Show more

 • Promoted

Data Engineer & Analytics

Kredivo GroupMarkham, York Region, CA
Full-time

As a Data Engineer and Analytic you will be tasked with empowering internal stakeholders to delve deep into the complexities of the data in our warehouse.You will work closely with Product, Enginee... Show more

 • Promoted

Data Engineer (AWS / Snowflake / DBT / Airflow / Informatica IICS)

act digitalToronto, ON, CA
Full-time

Data Engineer (AWS / Snowflake / DBT / Airflow / Informatica IICS).Act Digital is an international IT consulting firm with.Data & AI, Cybersecurity, Cloud & Infrastructure, and Software Delivery.We... Show more

 • Promoted

Data Engineer

Veeva Systems Inc.toronto, on, Canada
Full-time

We are looking for a talented and experienced data engineer to join our team.The ideal candidate will have a strong understanding of data engineering principles and practices, as well as experience... Show more

 • Promoted

Data Engineer

DS Technologies IncToronto, ON, CA
Full-time

Get AI-powered advice on this job and more exclusive features.About US: We are a company that provides innovative, transformative IT services and solutions.We are passionate about helping our clien... Show more

 • Promoted

Senior Data Engineer, Blockchain

CircleToronto, ON, CA
Full-time

Circle is committed to visibility and stability in everything we do.As we grow as an organization, we're expanding into some of the world's strongest jurisdictions.Speed and efficiency are motivato... Show more

 • Promoted

Data Engineer - Snowflake

Tiger AnalyticsToronto, ON, CA
Full-time

Tiger Analytics is a fast-growing advanced analytics consulting firm.Our consultants bring deep expertise in Data Science, Machine Learning and AI.We are the trusted analytics partner for several F... Show more

 • Promoted

Data Engineer

LuxoftToronto, ON, CA
Full-time

Get AI-powered advice on this job and more exclusive features.Direct message the job poster from Luxoft.You will work as a Data Engineer in Trading Technology, Public Market Investments team, partn... Show more

 • Promoted

Data Engineer

Quantum Management Services Ltd.Toronto, ON, CA
Temporary

Quantum Management Services Ltd.North York, ON (Hybrid – on-site Monday‑Thursday).We are looking for a Data Engineer to support large‑scale retail and e‑commerce data initiatives, working with comp... Show more

 • Promoted

Databricks Senior Engineer: Data-Driven Insights

WinsAboveToronto, ON, CA
Full-time

Accelerate data-driven transformation as a Senior Solutions Engineer at Databricks in Toronto, ON.Utilize your expertise in ML and AI to provide strategic guidance to large enterprises.In this infl... Show more

 • Promoted

Remote Lead Data Engineer - Databricks & PySpark

LumenaltaToronto, ON, CA
Remote
Full-time

A technology company is seeking an experienced Data Engineering Lead to guide technical direction for critical initiatives in data engineering.This fully remote position requires 10+ years of exper... Show more

 • Promoted

Data Engineer - DataBricks

Soar ConsultantsToronto, ON, CA
Full-time

We are currently looking for a Data Engineer for a client in Toronto.They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sector... Show more

 • Promoted

Lead Engineer, Data Enablement, Data Platforms

Canada Pension Plan Investment BoardToronto, ON, CA
Full-time

Make an impact at a global and dynamic investment organization**When you join CPP Investments, you are joining one of the world’s most admired and respected institutional investors.As a professiona... Show more

 • Promoted

Data Engineer

Canadian Tire CorporationToronto, ON, CA
Full-time

Advanced Analytics team at Canadian Tire Corporation, your primary focus will be on ensuring the stability, reliability, and performance of ML models in production environments.Working closely with... Show more