Senior Data Engineer

Overview

Join to apply for the Senior Data Engineer role at Equisoft .

Equisoft is a global provider of digital solutions for insurance and investment, with a comprehensive ecosystem of scalable solutions to help customers meet challenges of digital transformation. Our team is based in North America, the Caribbean, Latin America, Europe, Africa, Asia and Australia.

Role

The Senior Data Engineer reports to the AVP, Core Insurance and works closely with AI / ML teams, product managers, and software engineering teams. The incumbent will be responsible for designing, building, and maintaining robust data infrastructure and pipelines that support Equisoft's AI and ML initiatives, with a focus on scalable data processing and real-time analytics for insurance and investment solutions.

Responsibilities

• Build and maintain scalable Databricks pipelines for ML workflows, including data ingestion, transformation, and feature engineering for machine learning models
• Implement and optimize synthetic data generation infrastructure to support ML training while ensuring privacy compliance and data quality standards
• Create data augmentation pipelines designed for insurance scenarios, including policy data, claims processing, and risk assessment use cases
• Optimize data storage and retrieval systems for training efficiency, implementing partitioning strategies, caching mechanisms, and performance tuning
• Develop and maintain real-time stream processing capabilities using Apache Spark Structured Streaming, Kafka, and other modern streaming technologies
• Ensure comprehensive data quality and compliance for ML training datasets, implementing validation frameworks, monitoring systems, and governance policies
• Design and implement ETL / ELT pipelines using modern data stack tools including Apache Airflow, dbt, and cloud-native services
• Collaborate with ML teams to establish data versioning, lineage tracking, and reproducibility for model training datasets
• Monitor and troubleshoot data pipeline performance, implementing automated alerting and recovery mechanisms
• Work with cloud platforms (AWS, Azure, GCP) to architect scalable data solutions and optimize costs
• Implement data security best practices including encryption, access controls, and audit logging

Requirements

Technical

Soft Skills

Nice To Haves

Equal Opportunity

Equisoft is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status.

#J-18808-Ljbffr