Big Data Engineer

Who We Are

Teranet is Canadas leader in the delivery and transformation of statutory registry services with extensive expertise in land and commercial registries. We also market insightful property and data solutions as well as practice management automation to thousands of customers in the real estate financial services government utilities and legal markets. Founded in 1991 Teranet operates the Electronic Registration System for the Province of Ontario and Teranet Manitoba is the service provider to the Province of Manitoba.

Connect. Grow. Thrive Together.

To learn more about who we are visit our website:

About the Role

Teranet is seeking an experienced Data Engineer for its Enterprise Data Analytics team within the IT line of business. This is a pure data engineering role focused on processing large-scale data and optimizing Spark performance not a Data Analyst or BI Developer position.

In this role you will:

• Design build and orchestrate data pipelines using tools like Airflow.

• Integrate data from diverse sources such as RDBMS systems (via Debezium) and files (CSV JSON XML text) using Python.

• Be responsible for transforming and processing large volumes of data optimizing Spark execution plans for performance and cost efficiency and writing complex SQL and Python code.

• Work closely with product owners data analysts and other data engineers to develop and enhance data pipelines and data models that support data delivery AI development and BI insights.

• Configure data ingestion frameworks for databases like Oracle MS SQL Server and PostgreSQL while ensuring data accuracy quality and proper curation to meet various business use cases.

• Partner with infrastructure teams to enforce data security access controls and auditability ensuring that data is properly governed secure and compliant across the organization.

What Youll Be Doing

• Participate in planning with business product owners data analysts and identify tasks for the data analytics team.

• Design and develop data pipelines (Debezium and HVR) curate data for enterprise-wide usage prepare data models for specific use case.

• Develop test objectives test plan and success criteria (connectivity data replication auto fail-over peak load performance etc.).

• Work with infrastructure security and networking teams to ensure connectivity requirements are met for data pipelines sources and targets.

• Tuning of data ingestion and replication to meet performance targets.

• Configure the CDC framework as required to create daily/weekly/monthly data snapshots within acceptable performance targets.

• Design and implement technology best practices guidelines and repeatable processes. Create design test plan and confluence documentation.

• Able to self-direct prioritize and perform assignments with minimal supervision

About You

• 5 years of experience with modern data platform medallion architecture configuring data pipelines ETL RDBMS SQL Spark PySpark Bash Linux.

• Familiarity with AWS S3 Hive metastore Trino Airflow Control-M.

• Knowledge of CDC based data ingestion setup preferably using HVR and Debezium.

• Deep Hive/Spark/SQL knowledge development and testing experience

• Expert Python/Spark/Shell development and coding best practices skills

• Excellent day-to-day working knowledge of Git with exposure to Gerrit

• Extensive experience in developing ETLs and processing large datasets

• Experience with Airflow data pipeline orchestration

• Experience building data models to support BI data visualizations using Tableau

• Familiarity with AWS services such as S3 EKS and Kubectl is highly beneficial

• Excellent written and verbal communication skills

Lets Talk Pay

We believe in being upfront about pay and helping you make informed decisions about your career. The annual pay range for this role is $110000-$130000 inclusive of base salary and target incentive pay. We understand that great talent comes in many forms each with unique skills experience and potential. Your salary will be tailored to reflect the experience you bring and the impact youre ready to have on this role.

At Teranet we also know that compensation extends far beyond a pay cheque. Along with your cash compensation we offer a comprehensive package which includes the following:

• 100% Employer Paid Health Benefit Plan

• Employer Matching Retirement Savings Plan

• Paid Vacation Floater Days & Sick Leaves

• Maternity Parental and/or Adoption Leave Top-Up Programs

• Corporate Discounts & GoodLife Group Rate Membership

• Employee Assistance Program for you and your loved ones!

Our Human Approach to Hiring

At Teranet we use smart technology to make hiring faster and more efficient. Artificial Intelligence (AI) tools help us review applications and identify strong matchesbut the real decisions Those are made by our awesome people.

Every interaction youll have and every decision we make is led by humans. AI may support us but its human connection that drives every hire.

At Teranet we are committed to fostering an inclusive accessible environment where all employees and customers feel valued respected and supported. We are dedicated to building a workforce that reflects the diversity of our customers and the communities in which we live and serve. If you require accommodation during the recruitment and selection process please let us know and we will work with you to meet your needs.

Come As You Are. We Like You That Way

Required Experience:

IC

Employment Type : Full-Time
Experience: years
Vacancy: 1
Monthly Salary Salary: 110000 - 130000