Int Data Engineer to design, build and maintain data infrastructure
Our valued client is looking for an Int Data Engineer to design, build and maintain data infrastructure.
initial 3-6 month contract hybrid in Ottawa with possibility for conversion to full time permanent. (min 2 days a week onsite)
Responsibilities :
- Designing, building, and maintaining data infrastructure that supports the efficient extraction, transformation, and loading (ETL) of data from various sources.
- Develops ways to increase the usability of data through changes in business process and / or data cleansing techniques
- Design, build, and maintain data pipelines and ETL processes using tools such as Streamsets, Apache Spark, Hadoop, and other big data technologies.
- Develop and maintain data infrastructure including data warehouses and other data storage solutions to support business intelligence and reporting needs
- Design and implement data security measures to protect sensitive data.
- Develop and maintain data quality control processes to ensure the accuracy, reliability, and accessibility of data to all stakeholders.
- Monitors system performance of scheduled ETL batches, and streaming data process and ensures all systems are working at an acceptable standard and that data pipelines are scalable, repeatable, and secure.
- Performs data migration between development, UAT, and production systems, and plans and coordinates these data migrations.
- Analyzes and troubleshoots technical issues quickly to resolution, working with internal ITS sections and software vendors, when required.
Must Have Skills :
- 5+ years experience in a Data Analytics environment with progressively more technical responsibilities in an engineering role
- Designing and creating :
- ETL processes, data pipelines and workflows.
- Logical and physical data models using data modelling best practices.
- Develop scripts, applications and APISs to automate data processing tasks using programming languages such as SQL, Python, Java, Scala, shell scripting, JavaScript
- Designing, building, and supporting data warehouses, data hubs and other data storage and processing systems.
Nice to Have Skills :
- Experience with cloud computing platforms such as Azure and being familiar with setting up and managing cloud-based data storage and computing environments.
- Working with stream processing frameworks such as Apache Kafka or Streamsets
- Designing and implementing :
- Real-time data processing pipelines using these tools.
- Database solutions using technologies such as MySQL, PostgreSQL or SQL Server.
- Project implementation analysis and support in data management systems, data integrity and security as it relates to environmental business systems.
- Machine Learning concepts and tools (R, Python, Jupyter Notebook).
- Utilization of tools such as Apache Spark, Hadoop, and other big data technologies.
Data Engineer
Reporting to the Director of Data Management & Engineering the Data Engineer is experienced and passionate about building creative and novel data solutions. Develop and maintain scalable big data solutions, and data pipelines to support data transformation and analytics initiatives. By incorporating...
Lead Data Engineer
Kynetec is seeking an experienced Data Engineering leader who specialises in Microsoft SQL Server and Azure Databricks! You will be responsible for the day-to-day management of our Data Engineering Team in North America, and will support our client facing teams to optimize Kynetec’s data production ...
Senior Business Analyst to review data modela and develop process flows, work flows, and user stories for large-scale Fisheries Information System
Facilitate meetings and interviews to elicit business requirements, and pose questions to determine what the client’s needs are. Communicate and liaise with numerous stakeholders to gather information on information holdings and business requirements. Provide a coherent view of user stories so that ...
Records & Data Entry Assistant
The Records & Data Entry Assistant is responsible for data entry, maintaining the accuracy and integrity of computerized records systems and responding to requests and enquires. Ability to demonstrate attention to detail and accuracy. Ability to communicate with others to explain and share informati...
Data Engineer
Experience building external integrations and mapping data from one service in to another. Hands on experience in working with data warehouses and large amount of data. Minimal 5 years hands-on experience on building Data Pipelines. Minimal 5 years hands-on experience in Python, Java and SQL. ...
Senior Data Engineer
Engineer and enhance data pipelines, ETL processes, and data cleaning mechanisms to support a comprehensive cybersecurity data analysis system. We are on the lookout for a talented Senior Data Engineer to contribute significantly to our advanced data collection, processing, and analysis system which...
Senior Data Engineer
As a Senior Data Engineer, you will play a crucial role in designing, implementing, and optimizing our data infrastructure. Work closely with data engineering to design and implement data models that align with business requirements. Zonda is looking for a passionate Senior Data Engineer to evolve a...
Sr. Director, Demand Generation & Growth
Strong capability and passion designing integrated campaigns capable of creating awareness and capturing demand to a broad range of IT and engineering professionals. Director of Demand Generation & Growth is critical to the near- and long-term success of Redpanda. Reporting to the CMO, you will ...
Senior Data Scientist - Search & Recommendation
You’ll determine answers to questions like, how relevant is this product to this retailer and how can we infer that in real-time? Can we create real-time embeddings of users' interests and use them to power rankings? How can we improve purchase flow? What techniques can we use to enable our A/B test...
Payroll Automation & Data Expert
Provide vision and clear Business Requirement Documents (BRDs), to innovate and automate manual processes, to decrease inefficiencies and support with business growth. Work with Engineering, Product and Payroll to build intricate solutions to complex problems. Take a hands-on approach to implement s...