AI Engineer - Azure Databricks

Job Description

• Digital: Microsoft Azure
  

• Digital: Azure Machine Learning (ML)
  

---

Technical Must-Haves

• Hands-on experience with Unity Catalog governance and lineage.
  

• Strong experience with Azure services:
  

  • ADLS Gen2
    

  • Azure Data Factory (ADF) Pipelines
    

  • Azure Key Vault
    

  • Azure Event Hub / Service Bus
    

  • Azure DevOps
    

• Proven experience delivering RAG and agentic patterns, including:
  

  • Embeddings pipelines
    

  • Chunking strategies
    

  • Vector indexing
    

  • Evaluation frameworks
    

• Solid SQL expertise and strong understanding of:
  

  • Data modeling
    

  • Medallion Architecture (Bronze/Silver/Gold)
    

  • ACID transactions
    

• Proven experience implementing Model Context Protocol (MCP) for agent-tool integrations:
  

  • Building MCP servers/adapters to expose tools/resources (SQL, REST, files, APIs, etc.)
    

  • Defining tool schemas, input/output contracts, and safety guardrails
    

  • Securing MCP integrations using Azure AD (Entra ID), Key Vault, secret scopes, and network controls
    

  • Connecting MCP to Unity Catalog governed assets, Vector Search, and Model Serving
    

---

Essential Skills

1. AI / GenAI Engineering (MCP & RAG)

• Build GenAI-ready datasets, embeddings pipelines, and vector indexes for RAG workloads.
  

• Use Databricks Genie to accelerate:
  

  • Code creation
    

  • Optimization
    

  • Documentation
    

  • Refactoring
    

• Implement MCP patterns to connect AI agents to enterprise data and tools.
  

• Design, build, and operate MCP servers/adapters that expose tools such as:
  

  • SQL
    

  • REST APIs
    

  • File systems
    

  • Enterprise APIs
    

• Implement:
  

  • Tool schemas
    

  • Resource providers
    

  • Prompt and tool safety policies
    

• Integrate MCP with Azure Databricks workloads including:
  

  • Delta Tables
    

  • Unity Catalog
    

  • Vector Search
    

  • Model Serving
    

• Secure MCP endpoints using:
  

  • Key Vault
    

  • Secret scopes
    

  • Network rules
    

  • Role-based access control aligned with Unity Catalog
    

• Operationalize RAG and agentic workloads using:
  

  • Vector Search
    

  • Model Serving
    

  • Feature Store
    

  • MLflow
    

---

2. Azure Cloud Integration

• Integrate Azure Databricks with:
  

  • ADLS Gen2
    

  • ADF Pipelines
    

  • Key Vault
    

  • Event Hub / Service Bus
    

  • Azure Functions
    

• Implement telemetry and observability using:
  

  • Azure Monitor
    

  • Log Analytics
    

  • Databricks metrics
    

---

3. Data Governance & Architecture

• Implement Unity Catalog for:
  

  • Centralized governance
    

  • Data lineage
    

  • Fine-grained access controls across workspaces
    

• Apply Medallion Architecture (Bronze / Silver / Gold).
  

• Enforce data quality using:
  

  • Delta Live Tables
    

  • Expectations framework
    

• Manage:
  

  • PII/PHI controls
    

  • Data masking
    

  • Secure data sharing (e.g., Delta Sharing where applicable)
    

---

4. DevOps & Automation

• Build CI/CD pipelines using:
  

  • Azure DevOps
    

  • GitHub Actions
    

• Use Databricks Repos for version control.
  

• Orchestrate workloads using:
  

  • Databricks Workflows
    

  • Azure Data Factory
    

• Implement environment promotion paths (Dev → Test → Prod).
  

Requirements
Experience (Years): 8-10