Data Engineer

Job Description

Key Responsibilities

• Pipeline Development : Design, build, and optimize scalable data ingestion pipelines to extract, transform, and load (ETL) data from source systems (Oracle, SQL MI, MongoDB, file feeds) into Databricks.
• Medallion Architecture Implementation : Lead the adoption and implementation of Medallion architecture (Bronze, Silver, Gold layers) for data processing and reporting in Databricks.
• Data Modeling on Silver Layer : Develop and maintain effective data models within the Silver layer to support standardized, cleansed, and business-ready datasets. Ensure that data relationships, normalization, and transformation logic are well-defined to facilitate downstream analytics and reporting.
• Testing & Validation : Develop automated tests for data pipelines and transformations to ensure reliability and correctness.
• Data Quality & Governance : Ensure data quality, integrity, and consistency throughout the ingestion and transformation processes. Implement data validation, cleansing, and governance best practices.
• Collaboration : Work closely with data analysts, data scientists, and business stakeholders to understand requirements and deliver effective data solutions.
• Automation & Monitoring : Automate data pipeline workflows, monitor performance, and proactively resolve issues to ensure high reliability and availability.
• Documentation : Develop and maintain comprehensive documentation for pipelines, data flows, and architecture decisions.

Qualifications

Technical Skills :

Preferred :

Core Competencies :

#J-18808-Ljbffr