Senior Databricks Data Engineer

Salary: $100,000 - 160,000 per year

Requirements:

• 4+ years of data engineering experience, including at least 2 years working hands-on with Databricks
• Strong Python skills and advanced SQL capability
• Solid PySpark experience and a good grasp of distributed processing, including troubleshooting and tuning Spark jobs
• Production experience with Delta Lake and medallion architecture (bronze, silver, gold)
• Cloud experience on Azure (ADLS Gen2, Data Factory, Azure DevOps) or AWS (S3, Glue, Lambda)
• Comfortable using Git-based workflows and CI/CD for data pipelines
• Experience leading workshops and requirements sessions, not just attending them
• Proven experience working directly with business stakeholders rather than through a project manager or BA
• Strong elicitation skills, including knowing which questions to ask and when to keep probing
• Clear written communication, including requirements documentation, process maps and design documents
• Confidence presenting to senior stakeholders and standing your ground when challenged
• Commercial awareness to distinguish between what a client requested and what will genuinely add value
• Comfort working in ambiguous early-stage engagements where requirements are not yet fully defined
• Legally entitled to work in Canada
• Databricks certification, Data Engineer Associate or Professional, is nice to have
• Experience with Unity Catalog rollouts and data governance is nice to have
• Experience with Lakeflow Declarative Pipelines / Delta Live Tables, structured streaming, or Databricks Asset Bundles is nice to have
• Experience with Terraform or similar infrastructure-as-code tools is nice to have
• Experience with dbt or Airflow is nice to have
• Experience with Power BI, Tableau or similar is nice to have
• Experience with MLflow or supporting ML and GenAI workloads on Databricks is nice to have
• Prior consulting or professional services experience is nice to have
• Presales or solution estimation exposure is nice to have
• Experience in energy, utilities or resources is nice to have

Responsibilities:

• Run discovery sessions and requirements workshops with stakeholders who may not yet be able to clearly define what they need
• Dig beneath the stated request to uncover the underlying problem, and challenge it when it does not align
• Map current processes, data flows and pain points, then document requirements and acceptance criteria clearly
• Translate business issues into technical designs, and technical limitations into language that non-technical executives can act on
• Manage scope and expectations across an engagement, including pushing back when needed
• Present designs, trade-offs and progress updates to audiences ranging from data teams to executive sponsors
• Build strong client relationships that encourage repeat business and reveal further opportunities
• Design, build and optimise data pipelines on Databricks using PySpark, Spark SQL and Python
• Implement lakehouse architectures using Delta Lake, Unity Catalog, medallion patterns and Databricks Workflows
• Migrate legacy warehouses, older ETL tools and on-premises workloads to Databricks
• Build ingestion from APIs, streaming sources, relational databases, flat files and SaaS platforms
• Apply solid engineering practices to data delivery, including version control, CI/CD, automated testing and infrastructure as code
• Set up data governance, lineage, access control and data quality monitoring
• Tune performance and manage cost across clusters, jobs and SQL warehouses
• Mentor junior engineers and contribute to the direction of the data practice

Technologies:

• AI
• Airflow
• AWS
• Lambda
• Azure
• CI/CD
• Cloud
• Databricks
• DevOps
• ETL
• Git
• Support
• MLflow
• Power BI
• Project Manager
• Python
• PySpark
• SQL
• Spark
• Tableau
• Terraform
• UiPath
• Unity
• dbt
• GameDev
• ARM

More:

We are Blackbook AI, a technology consultancy focused on AI, automation, data and digital. Founded in Australia, our head office is in Brisbane, and over the past nine years we have expanded across APAC and North America, including a growing Calgary team. We have delivered more than 2,000 projects for over 180 clients, and we are a Databricks Consulting Partner as well as a Microsoft Solutions Partner, AWS Consulting Partner and UiPath Diamond Partner. We hire locally because people who understand the market tend to do better work in it, and we support them with a global delivery team behind the scenes. This is a full-time, in-person Calgary role with a salary range of $100,000 to $160,000 per year, offering variety across multiple industries, close involvement in discovery and consulting, ongoing certifications and training, and the chance to help shape a growing Canadian practice.

last updated 38 week of 2026