RQ10463 - Software Developer - ETL - Senior

Scope of Services and Deliverables - The Services and Deliverables to be provided by the Vendor will include the following:

• Participate in the review of project, work request and maintenance release artefacts - Use Case, System Requirement Specification, etc.
  
• Deliver detailed system design documents and any other supporting documentation conforming to Ministry standards.
  
• Design logical and physical data models using Sybase Power Designer.
  
• Produce transformation mapping document.
  
• Implement data pipelines using Azure Data Factory (ADF), stored procedure, and Informatica PowerCenter performing extraction, transformation, and loading activities.
  
• Implement solutions using Logic App and Function App.
  
• Generate structured JSON file using ADF/ SQL.
  
• Develop Azure CI/CD pipeline to automate ADF release.
  
• Develop complex Oracle PL/SQL program to fulfill project requirements.
  
• Implement complex data conversion, e.g., binary to character, EBCDIC to UTF.
  
• Implement complex data transformation(s) for derived and calculated values.
  
• Integrate data sets from diverse source systems.
  
• Implement ADF for initial data load and incremental load.
  
• Promote ADF and Informatica ETL/ELT through all Ministry environments including Development, Integration Testing, QA, UAT, Production.
  
• Resolve and troubleshoot ADF pipeline, stored procedure, and Informatica workflow technical problems.
  
• Monitor and analyze incident, provide timely resolution.
  
• Collaborate with IT Professionals throughout the Software Development Life Cycle.
  
• Document information from diverse business area stakeholders and subject matter experts.
  
• Optimize performance of ADF pipeline, Azure SQL and Synapse databases, and Informatica workflow.
  
• Monitor application functionality and performance on daily basis.
  
• Provide effective knowledge transfer to Ministry staff at all stages of this assignment.
  

Requirements

ETL Development-Data Integration 40%

Extensive experience in gathering requirements and business process knowledge in order to design correct and high-quality data transformation.

Extensive experience in design, development, and implementation with Azure Data Factory, Databricks, including:

• Self-Hosted Integration Runtime
  
• Data movement, ETL pipelines
  
• Performance tuning on large data volumes
  
• Complex data transformations
  
• ADF Debugger to validate data transformation.
  
• Azure Roles
  

Experience with Informatica, including:

• Workflow Manager
  
• Repository Manager
  
• Designer
  
• Workflow Monitor
  
• Experience with developing solutions using Logic App and Function App.
  
• Experience in rapid application development (RAD) methodologies.
  
• Experience in maintaining and improving existing ETL processes.
  
• Experience in investigating data to identify potential issues within ETL pipelines.
  
• Experience with Business Intelligence tools IBM Cognos and Microsoft PowerBI.
  

Azure Technologies 40%

Experience with CI/CD (DevOps) pipelines and concepts, including:

• Azure Resource Management
  
• GitHub Repo
  
• Source code version control
  
• Branching, pull requests, build, release, multiple parallel development repositories merge.
  

Experience with Azure DevOps, including:

• Azure Boards
  
• Azure Test Plans
  
• Azure Pipelines
  
• Azure Repos
  
• Azure Artefacts
  

Oracle PL/SQL Development 10%

• Extensive experience in designing and developing SQL Server Stored Procedure, Oracle PL/SQL programs.
  
  

Database Technologies 10%

• Experience with Oracle, Microsoft SQL Server database and tools.
  
• Experience with relational and hierarchical database technologies.
  
• Extensive knowledge of Azure SQL DB and Synapse platforms, including:
  
• Performance tuning on large data volume
  

Must have:

• Extensive experience in design, development, and implementation with Azure Data Factory, Databricks
  
• Experience with CI/CD (DevOps) pipelines and concepts
  
• Extensive experience in designing and developing SQL Server Stored Procedure, Oracle PL/SQL programs.
  
• Extensive knowledge of Azure SQL DB and Synapse platforms
  

• Experience with Informatica