Data Engineer
Data Engineer - #61261
About the Job :
Mindwire is currently seeking a Data Engineer to work for our valued Crown corporation client.
The position is remote with the option of a Hybrid work environment (Ottawa / Toronto).
Responsibilities :
- Converts historical data from SAS database to Parquet.
- Works closely with teams to develop and transition ongoing production process of analytical assets (currently coded in SAS and stored as SAS datasets) to be coded using Python (leveraging Pyspark) and stored in Parquet.
- Recommends improvements and enhancements to conversion processes and ongoing production of analytical assets.
- Prepares documentation required related to the results of the conversion (e.g. differences in converted data that will impact use by analysts).
- Coaches and mentors staff in data conversion techniques.
Requirements and Qualifications :
- 5 years’ experience with version control using Git, including branching, merging, and collaborating with teams to manage code repositories effectively.
- Extensive knowledge of SQL for querying and manipulating databases, with a solid understanding of database design and optimization (5 Years’ Experience).
- High level of expertise in data engineering processes, including data extraction, transformation, and loading (ETL), utilizing Informatica tools to create efficient and robust data pipelines.
- Extensive programming experience in Python, with the ability to develop complex applications, scripts, and data processing tasks using a wide range of libraries and frameworks.
- Proficient usage of Jupyter Notebooks as an integrated development environment for data analysis, data visualization, and interactive code execution, fostering efficient and reproducible workflows.
- Extensive hands-on experience with Apache Spark, including the ability to design, optimize, and manage large-scale data processing and analytics pipelines.
- In-depth knowledge of Apache Parquet file format for columnar storage, including its benefits and optimal usage in big data processing scenarios.
Data Engineer - Flink
Spark expert, exp with Scala, Big Data, Flink. ...
Forcepoint DLP (Data Loss Prevention) Engineer
Forcepoint DLP (Data Loss Prevention) Engineer. CASB, Data Classification, Data Discovery, encryption, etc. Forcepoint DLP Engineer or Forcepoint NGFW Engineer). This qualified individual will provide engineering, installation, configuration and integration of Forcepoint products. ...
Data Analytics Engineer II (Canada Remote)
Using that insight, you will write the data transformation code in ELT pipelines to integrate that data into our data warehouse for consumption by the reporting and data analytics functions. Working with other data engineers you will create and populate the data models for downstream consumption and...
Staff Data Engineer
As a Staff Data Engineer, you will contribute to enhancing and maintaining our data processing and storage pipelines/workflows for a robust and secure finance data lake. You should have a strong engineering background and even more importantly a desire to take ownership of our data systems to make t...
Data Engineer II
Your expertise in Databricks will be essential in optimizing data workflows, ensuring data quality, and contributing to the continuous improvement of our data engineering practices. Implement best practices for data modeling, ETL processes, and data ingestion, and enhance data engineering practices....
Senior Data Engineer, Implementation Lead
Datavant is a data logistics company for healthcare whose products and solutions enable organizations to move and connect data securely. We are a data logistics company for healthcare whose products and solutions enable organizations to move and connect data securely. Datavant has a network of netwo...
Lead Data Engineer
Lead an engineering team to design, code, and test cloud-based data pipelines to process and store large-scale datasets. Ipsos Global Data Management Tower serves as the backbone of Ipsos’ efforts to harness and integrate the power of data, globally. We’re looking for someone with a deep interest an...
Staff Engineer, Audiences & Customer Data Platform
We're seeking a Staff Engineer to help lead our growing full-stack engineering team. We are also building a customer data platform that will help to organize and manage audience data from many sources. As an Engineer at StackAdapt, you will be directly involved in the development of our advertising ...
Coop Student, Data Engineer
Enrolment in a recognized post-secondary co-operative program in computer science, computer engineering, information technology or related field. ...
Data Engineer
WestJet Group is seeking a Data Engineer for the progressive, essential, and fun Data Engineering team. Advise on the right way to apply data engineering standards and methods, ensuring compliance, and maintain technical responsibility for all the stages and iterations of a data engineering project....