Python Data Platform Engineer

Job Duties :

As a Python Data Platform Engineer, you will be a member of the C3 Data Warehouse team within the Controls Engineering, Measurement and Analytics (CEMA) department, with a focus on building our next-gen data platform used for sourcing and storing data from different technology systems across the firm into a centralized data platform that empowers various reporting and analytics solutions for the Technology Risk functions within Morgan Stanley. In this role you will be primarily responsible for contributing to the development of a unified data pipeline framework written in Python utilizing technologies such as Airflow, DBT, Spark and Snowflake. You will also be responsible for contributing to the integration of this framework with existing internal platforms for data quality, data cataloging, data discovery, incident logging, and metric generation. You will be working closely with data warehousing leads, data analysts, ETL developers, infrastructure engineers, and data analytics teams to facilitate the implementation of this data platform and data pipeline framework.

KEY RESPONSIBILITIES :

• To develop various components in Python of our unified data pipeline framework.
• To contribute towards the establishment of best practices for the optimal and efficient usage of Airflow, DBT and Snowflake.
• To assist with the testing and deployment of our data pipeline framework utilizing standard testing frameworks and CI / CD tooling.
• To monitor the performance of queries and data loads and perform tuning as necessary.
• To provide assistance and guidance during the QA & UAT phases to quickly confirm the validity of potential issues and to determine the root cause and best resolution of verified issues.

Minimum Skills Required :

#J-18808-Ljbffr