ETL Developer 7228-1510

HM Note : This hybrid role isthree (3) days in office and candidates are required to providethree references in theirresume.

Description :

GeneralResponsibilities

Designdevelop and implement ingestion framework from Oracle data sourceto Azure Data Lake initial load and incremental ETL. Tools usedare :

Azure Data Factory(good knowledge required) to maintain pipeline from Oracle to AzureData Lake

AzureDatabricks / PySpark (good Python / PySpark knowledge required) tobuild transformations of raw data into curated zone in the datalake

AzureDatabricks / PySpark / SQL (good SQL knowledge required) to developand / or troubleshoot transformations of curated data into datamartmodel

• Reviewthe requirements database tables and database relationshipsIdentify gaps and inefficiencies in current production reportingenvironment and provide recommendations to address them in the newplatform.
• Designingestion framework and CDC tools used are Oracle Golden Gate andAzure DataFactory
• Prepare designartifacts
• Work with ITpartner on configuration of Golden Gate responsible to providedirection and howto.
• Maintain dynamicpipeline for ETL ingestion to add new tables and dataelements
• Data designphysical model mapping from data source to reportingdestination.
• Understandthe requirements. Recommend changes to the physical model tosupport ETLdesign.
• Reverseengineer and document existing SQL logic to improve designeffort
• Assist withdata modelling and updates of sourcetotarget mappingdocumentation
• Developscripts for the physical model and update database and / or data lakestructure.
• AccessOracle DB SQL Server and Azure environments and nbsp;using andnbsp; SSIS SQLDeveloper Azure Data StudioAzure Data Factory Databricks and other andnbsp;

tools to developsolution.

• Proactivelycommunicate with business and IT experts on any changes required toconceptual logical and physical models communicate and reviewtimelines dependencies andrisks.
• Development ofETL and nbsp; strategyand andnbsp; solution for different sets of datamodules
• Understand theTables and Relationships in the datamodel.
• Create lowlevel design documents and test cases for ETLdevelopment.
• Createthe workflows and pipelinedesign
• Development andtesting of data pipelines with Incremental and FullLoad.
• Develop highquality ETLmappings / scripts / notebooks
• Developand maintain pipeline from Oracle data source to Azure Data Lakeand Databricks SqlWarehouse
• Develop ETLto update datamarts built in and nbsp;Databricks SqlWarehouse
• Perform unittesting
• Ensureperformance monitoring andimprovement
• Performancereview data consistencychecks
• Troubleshootperformance issues ETL issues log activity for each pipeline andtransformation.
• Reviewand optimize overall ETLperformance.
• Endtoendintegrated testing for Full Load and IncrementalLoad
• Plan for Go LiveProductionDeployment.
• Createproduction deploymentsteps.
• Configureparameters scripts for go live. Test and review theinstructions.
• Createrelease documents and help build and deploy code acrossservers.
• Go LiveSupport and Review after GoLive.
• Review existingETL process tools and provide recommendation on improvingperformance and reduce ETLtimelines.
• Reviewinfrastructure and remediate issues for overall processimprovement
• KnowledgeTransfer to Ministry staff development of documentation on the workcompleted.
• Documentwork and share the ETL endtoend design troubleshooting stepsconfiguration and scriptsreview.
• Transferdocuments scripts and review of documents toMinistry.

Skills

Experienceand Skill SetRequirements

Experience :

• Experienceof 7 years of working with and nbsp; SQLServer TSQL Oracle PL / SQL development or similar relationaldatabases(musthave)
• Experienceof 2 years of working with and nbsp; AzureData Factory Databricks and Python development(musthave)
• Experiencebuilding data ingestion and change data capture using Oracle GoldenGate(nicetohave)
• Experienceworking with building databases data warehouses and dimensionaldata marts and working with delta and full loads(musthave)
• Experienceon Data modeling and tools e.g. SAP Power Designer Visio or similar(musthave)
• Experiencewith dimensional modeling. Experience in designing data warehousesolutions using slowly changing dimensions(musthave)
• Experienceworking with SQL Server SSIS or other ETL tools solid knowledge andexperience with SQL scripting(musthave)
• Experiencedeveloping in an Agileenvironment
• Understandingdata warehouse architecture with a delta lake and dimensional model(musthave)
• Ability toanalyze design develop test and document ETL pipelines fromdetailed and highlevel specifications and assist introubleshooting.
• Abilityto utilize SQL to perform DDL tasks and complexqueries
• Good knowledgeof database performance optimizationtechniques
• Ability toassist in the requirements analysis and subsequentdevelopments
• Abilityto conduct unit testing and assist in test preparations to ensuredata integrity
• Workclosely with Designers Business Analysts and otherDevelopers
• Liaise withProject Managers Quality Assurance Analysts and BusinessIntelligenceConsultants
• Design andimplement technical enhancements of Data Warehouse asrequired.

Skills :

• 7years using ETL tools such as Microsoft SSIS stored procedures TSQLand nbsp; (MustHave)
• 2 Azure DataLake and Databricks and building Azure Data Factory and AzureDatabricks pipelines and nbsp;(MustHave)
• 2 years Pythonand PySpark and nbsp;(MustHave)
• Oracle GoldenGate
• SQLServer
• Oracle
• Abilityto present technical requirements to thebusiness

Assets :

Knowledgeand experience building data ingestion history change data captureusing Oracle Golden Gate is anasset.

EvaluationCriteria

DesignDocumentation and Analysis Skills (30points)

• Demonstratedexperience in creating both Functional Design Documents (FDD) andamp; Detailed Design Documents(DDD).
• Experience inFitGap analysis system use case reviews requirements reviews codingexercises andreviews.
• Experience inthe development and maintaining a plan to address contractdeliverables through the identification of significant milestonesand expected results with weekly statusreporting.
• Work withthe Client and amp; Developer(s) assigned to refine / confirmBusinessRequirements
• Participatein defect fixing testing support and development activities for ETLand reporting
• Analyzeand nbsp;and document solution complexity and interdependencies byfunction including providing support for datavalidation.

DevelopmentDatabase and ETL experience (60points)

• Demonstratedexperience in database and ETL development (7years)
• Experience indeveloping in an agile Azure DevOpsenvironment
• Experiencein application mapping to populate data warehouse and dimensionaldata martschemas
• Demonstratedexperience in Extract Transform and amp; Load software development(7 years)
• Experiencein providing ongoing support on Azure pipeline / configuration andSqlServer SSIS andnbsp;development
• Experiencebuilding data ingestion and change data capture using GoldenGate
• Assist in thedevelopment of the predefined and adhoc reports and meet the codingand accessibilityrequirements.
• Demonstratedexperience with Oracle and SqlServerdatabases
• Proficientin SQL andPython
• Implementinglogical and physical datamodels

KnowledgeTransfer (10points)

TheDeveloper must have previous work experience in conductingKnowledge Transfer and training sessions ensuring the resourceswill receive the required knowledge to support the system.

Theresource must develop learning activities using reviewwatchdomethodology and amp; demonstrate the ability to prepare andpresent.

• Developmentof documentation and materials as part of a review and knowledgetransfer to othermembers
• Developmentand facilitation of classroom based or virtual instructor led demosessions forDevelopers
• Monitoridentified milestones and submission of status reports to ensureKnowledge Transfer is fullycompleted
30+ days ago