Data Engineering Testing Architect

Role: Data Engineering Testing Architect

Location: Halifax, CA (Remote)

Type: Contract

Job Summary:

• The Data Engineering Testing Architect is a senior role responsible for defining and driving the enterprise-wide testing strategy across modern data platforms, including data lakes, data warehouses, and data pipelines. This role ensures the reliability, scalability, and integrity of data systems by establishing robust validation frameworks, automation practices, and governance standards.
• The architect works closely with data engineering, analytics, and business teams to ensure that data assets are accurate, consistent, and fit for purpose across the organization, while also enabling modern, scalable data quality and observability practices.

Key Responsibilities:

• Define and implement an enterprise-level testing strategy for data platforms
• Establish and own data quality, validation, and governance frameworks
• Design testing approaches for ETL/ELT pipelines, batch processing, and streaming systems
• Enable and scale test automation across big data ecosystems (e.g. Databricks, Snowflake)
• Define standards for data reconciliation, lineage validation, and schema testing
• Implement and drive data observability and data profiling practices to proactively monitor data health
• Perform current state assessments of data testing and quality frameworks and define transformation roadmaps
• Propose and implement modern data testing solutions and best practices aligned with industry standards
• Partner with data/QA engineers, analysts, and business stakeholders to ensure data accuracy and usability
• Drive adoption of testing best practices and shift-left approaches within data engineering teams
• Ensure adherence to regulatory and compliance requirements and maintain data integrity
• Lead defect analysis and continuous improvement of data quality and pipeline reliability
• Build solution offerings, accelerators, and reusable frameworks for data testing and quality engineering
• Create and deliver client presentations, proposals, and solution narratives to support business development and win engagements

Required Skills & Qualifications:

• 10 15+ years of experience in data engineering, QA, or data validation/testing
• Strong hands-on expertise in SQL and Python
• Experience working with large-scale data platforms and distributed systems
• Proven experience with data testing frameworks (e.g., Great Expectations )
• Solid understanding of data modeling concepts (dimensional modeling, normalization, Lakehouse patterns)
• Knowledge of data governance, metadata, lineage, and data quality principles
• Experience with data observability and profiling tools and techniques
• Experience with cloud platforms such as Azure or AWS
• Familiarity with CI/CD and automation practices in data pipelines
• Basic familiarity with AI-assisted development tools or intelligent data quality techniques (e.g., anomaly detection, pattern-based validations)
• Excellent client communication and presentation skills, with the ability to articulate complex data quality and testing concepts to both technical and non-technical stakeholders
• Demonstrated ability to perform current state assessments and recommend scalable, modern solutions
• Strong problem-solving and stakeholder management skills

Preferred Skills:

• Experience with data mesh or Lakehouse architectures
• Exposure to real-time data processing frameworks (e.g., Kafka, Event Hubs)
• Understanding of DevOps/DataOps practices
• Experience in performance and scalability testing of data systems

Tools & Technologies:

• Data Platforms: Databricks, Snowflake
• Processing Frameworks: Apache Spark, Hadoop
• Testing Frameworks: Great Expectations
• Programming & Querying: Python, SQL
• Orchestration: Airflow, Azure Data Factory, Prefect
• Cloud: Azure, AWS, GCP
• CI/CD: Jenkins, GitHub Actions, Azure DevOps
• Streaming: Kafka, Azure Event Hubs
• Governance & Lineage: Collibra, Alation, Apache Atlas

Leadership Expectations:

• Provide architect-level ownership of data testing practices across the organization
• Define and enforce standards, frameworks, and best practices
• Mentor and guide engineering and QA teams
• Drive alignment across teams and influence data quality and reliability initiatives
• Contribute to capability building, solution development, and pre-sales support
• Promote a culture of accountability and quality in data delivery

KPIs / Success Metrics:

• Improvement in data quality and accuracy
• Increased reliability and stability of data pipelines
• Reduction in data defects and production issues
• Growth in test automation coverage
• Adoption of data observability and profiling practices
• Faster and more predictable data validation cycles
• Contribution to solution offerings and successful client engagements
• Compliance and audit readiness
• Good to Have
• Prior experience working in the healthcare payer domain, with understanding of claims, membership, and provider data
• Familiarity with healthcare data standards and regulatory considerations