Senior Data Engineer

Data Engineer with Databricks & Spark Experience

We are looking for a data engineer who has strong experience in building scalable and reliable data pipelines using Databricks and Spark.

You will be working with various data sources and formats, and transforming them into valuable insights for our business.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed
• Optimize data pipelines for performance, scalability, and reliability
• Ensure data quality and integrity throughout the data lifecycle
• Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs
• Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations
• Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management
• Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community
• Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent work experience
• 5+ years of experience in data engineering with Databricks and Spark
• Proficient in SQL and Python, and familiar with Java or Scala
• Experience with cloud platforms, such as Azure or AWS
• Experience with big data technologies, such as Kafka, Hadoop, Hive, etc.
• Experience with data warehouse and data lake concepts and architectures
• Experience with data integration and ETL tools, such as Azure Data Factory or Talend
• Experience with data visualization and reporting tools, such as Power BI or Tableau
• Strong analytical and problem-solving skills
• Excellent communication and teamwork skills
12 days ago