Principal Data Engineer (Hadoop, Spark, Flink, Kafka)

We shaped the earliest forms of ad tech, and we’re looking for technical expertise to help shape its future. Our customers have unique problems that can only be solved at an internet scale, and that’s where the technical skills of our team make a real difference.

Our exchange handles over 350 billion requests every day (for comparison Google serves an estimated 9 billion searches a day), all running in our own global data centers.

Every member of our technology team has an enormous amount of autonomy in building and managing our systems to support and enable our growing level of scale.

Through the transparency of our technology, dedication to innovation and integrity, and long-standing customer relationships, we lead through change.

What’s it like to work at Index?

We have more than 550 Indexers around the globe dedicated to building a safe and transparent marketplace that provides a trusted experience for consumers.

Index is an exciting and fast-paced place to work. We’re built on our values of change, support, learning and teaching, trust, and intention.

We pride ourselves on our independence and openness, not only in our technology but in our teams, too. Our diverse and inclusive culture celebrates how we can leverage our unique differences to help drive Index forward.

Our culture of success is truly supportive and collaborative. In working together across our teams, we’re continually investing in the people and technology to solve the industry’s most complex problems.

As we extend the promise of ad tech to every channel, we’re looking for talented engineers to help advance Index, and the industry, forward.

Are you ready to join the programmatic evolution?

Index Exchange funds the open web. Content and journalism across the internet are funded through advertising, and we are the engine that helps to make that happen transparently, safely, and efficiently.

Handling hundreds of billions of auctions per day within milliseconds requires an intense understanding of the exchange and the ecosystem that we live in.

Our business is growing significantly every year and is poised to grow even faster. Our people and our platforms are the foundation and enablers of that growth.

We are significantly expanding our technology teams, and are looking for technologists with a passion for high-performance software development, and a drive to deliver software products and platforms that enable and empower industries at a global scale.

About The Role

Join our global organization at the forefront of innovation in the Ad-Tech industry. We are seeking a dynamic and experienced Principal Data Engineer (Hadoop, Spark, Flink, Kafka) to lead across data domains and teams, driving strategic outcomes for our organization.

The ideal candidate will be equal parts responsible for driving initiatives at deep technical level and ensuring robust operational rigor to achieve stability and scale in our data environments.

Here’s What You’ll be Doing

• Leadership Across Domains :
• Provide strategic leadership across multiple domains, collaborating with cross-functional teams such as product, operations, infrastructure, and platform engineering to achieve business objectives.
• Foster a culture of innovation, continuous improvement, and collaboration within the data systems team.
• Advocate for strategic directions that reduce our complexity and tech stack footprint.

Technical Depth :

• Demonstrate deep expertise in our core data streaming components, including Kafka, Spark, Hadoop, Presto / Trino, Flink and Redis in a low latency environment.
• Architect relational data structures for performance, scale, and continuity.
• Drive technical excellence by staying abreast of industry trends, evaluating emerging technologies, and making recommendations for adoption.
• Lead initiatives with other cross-functional Principal engineering staff that are supporters and consumers of your data systems.

Operational Rigor :

• Ensure robust operational processes are in place to guarantee stability and scalability of database environments.
• Develop and implement best practices for database administration, monitoring, and maintenance using open-source observability and instrumentation tools.
• Develop and enhance testing and production release procedures with an automation-first core principle to end-to-end delivery.

Collaboration and Communication :

• Collaborate effectively with cross-functional teams, including developers, system administrators, and data scientists, to optimize database performance and functionality.
• Communicate complex technical concepts to non-technical stakeholders, fostering understanding and alignment across the organization.
• Participate in quarterly planning events to present your roadmaps and ensure strong alignment with the objectives and the broader technology teams.

Problem Solving and Troubleshooting :

• Lead efforts in problem identification, troubleshooting, and resolution to minimize downtime and enhance our Exchange reliability.
• Lead retrospective / rca engagements for data systems to ensure we are on a continuous improvement path.

Technology Stack :

• Deep understanding of streaming data components, including Kafka, Spark, Hadoop, Presto / Trino, Flink and Redis.
• Possess strong working knowledge of Ansible, Terraform, Kubernetes, Grafana, Prometheus, ELK
• Exposure and understanding of on-premise data center operations
• Exposure to other key database technologies is a plus (e.g. MySQL, MariaDB, Galera, PostgreSQL, Vertica, Aerospike).

Here's What You Need :

• 3-5 years of proven experience as a Principal Engineer or similar role.
• In depth expertise with streaming data components Kafka, Spark, Flink
• Hadoop, Presto / Trino, Redis
• Demonstrated leadership, mentoring, and collaboration skills balancing strategic and tactical needs.
• Excellent problem-solving and troubleshooting abilities, diagnosing at depth for a globally distributed infrastructure with low-latency requirements.
• Effective communication skills, both technical and non-technical audiences

If you are a seasoned Data Systems Engineer with a passion for driving strategic outcomes and ensuring technical excellence, we invite you to apply and be a key player in shaping the future of our global data environments.

Why You’ll Love Working Here :

• Comprehensive health, dental, and vision plans at no cost to you
• Time off and flexible work schedules
• Retirement plan with a 5% company match
• Stock options and equity packages
• Generous parental leave
• Monthly wellness stipend plus fitness discounts and quarterly wellness group activities
• Community engagement opportunities and donation-matching program
• Annual virtual company retreats and regular community-led team events
• One day off per year to volunteer
• A workplace that supports a diverse, equitable, and inclusive environment learn more here

Notification

Index Exchange is aware that there have been recent scams directed toward candidates regarding job interviews and offers.

Please be vigilant and do not accept interview requests, job offers, or other hiring-related documents from anyone other than our dedicated recruitment team, from the domain of @indexexchange.

com. Our interview process consists of several steps, including phone screens and video interviews. We do not conduct interviews via an email questionnaire or request money at any point in the process.

If you do receive these requests, please let us know immediately at : [email protected].

Equal employment opportunity

At Index Exchange, we believe that successful products are built by teams just as diverse as the audience who uses them.

As such, we are committed to equal employment opportunities. We celebrate diversity of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or expression, or veteran status.

Additionally, we realize that diversity is deeper than any status or classification diversity is the human experience. For those who show grit, passion, and humility Index will welcome you.

Accessibility for applicants with disabilities

Index Exchange is committed to working with and providing access and reasonable accommodations to applicants with disabilities.

Please let us know if you’d like to request a reasonable accommodation.

Index Everywhere, Index Anywhere

Our corporate headquarters are in Toronto, with major offices in New York, Montreal, Kitchener, London, San Francisco, and many other global cities.

As a major global advertising exchange, we are committed to operating as a tightly knit global team and embracing and empowering talent wherever our colleagues may be.

LI-ONSITE

LI-LP1