Databricks Consultant (ID#5473)

New Value Solutions, a national IT consulting company, is seeking a Databricks Consultant to design, lead implementation, and operationalize a secure, scalable Databricks foundation, with a strong focus on Unity Catalog, data governance, platform security, workspace architecture, storage integration, identity and access management, and production-ready data engineering patterns.

The successful candidate will work with data engineering, cloud infrastructure, security, governance, and business teams to establish an enterprise-grade Databricks environment that supports analytics, machine learning, data products, and governed self-service access.

Unity Catalog is Databricks’ unified governance layer for data and AI assets, providing centralized access control, lineage, discovery, auditing, and governance capabilities across Databricks workspaces

Responsibilities:

Databricks Foundation Implementation

• Design and lead implementation of the core Databricks platform foundation across development, test, and production environments.
• Define workspace architecture, naming standards, environment separation, identity model, network design, and workspace lifecycle management.
• Configure Databricks account-level settings, workspaces, groups, service principals, credentials, access policies, and administrative controls.
• Establish best practices for cluster policies, SQL warehouses, jobs, workflows, repos, secrets, and compute governance.
• Lead Implementation of platform standards for scalability, cost control, monitoring, operational support, and production readiness.

Unity Catalog Design and Implementation

• Design and deploy Unity Catalog as the enterprise governance layer for Databricks.
• Define the Unity Catalog metastore strategy, catalog, and schema structure, naming conventions, ownership model, and environment separation.
• Configure catalogs, schemas, tables, views, volumes, external locations, storage credentials, and permissions.
• Implement fine-grained access control using Unity Catalog privileges, groups, service principals, row filters, column masks, and attribute-based access controls where applicable.
• Establish governed data access patterns for data engineers, analysts, data scientists, and business users.
• Enable data discovery, lineage, auditability, and data asset management using Unity Catalog capabilities. Unity Catalog supports access control, discovery, lineage, auditing, governed tags, data classification, quality monitoring, data sharing, and AI governance

Cloud Storage and Security Integration

• Design secure integration between Databricks and cloud storage platforms such as ADLS Gen2, AWS S3, or Google Cloud Storage.
• Configure storage credentials, external locations, managed storage, and secure access patterns.
• Work with cloud infrastructure and cybersecurity teams to implement networking, private endpoints, encryption, firewall rules, secrets management, and least-privilege access.
• Ensure alignment with enterprise security, privacy, compliance, and data governance requirements.
• Define recommended patterns for Delta Lake, medallion architecture, streaming, batch processing, and data product development.
• Support migration of existing tables, data pipelines, notebooks, and workflows to Unity Catalog-enabled architecture.
• Advise teams on managed versus external tables, external volumes, schema design, permissions, and lifecycle management.
• Create reusable templates, standards, and implementation guides for Databricks development teams.

Governance, Operations, and Documentation

• Document Databricks platform architecture, Unity Catalog design, security model, operational procedures, and support processes.
• Develop runbooks for workspace administration, access provisioning, incident response, job monitoring, and cost management.
• Define roles and responsibilities for platform administrators, data owners, data stewards, engineers, and consumers.
• Support knowledge transfer and training for internal teams.
• Provide recommendations for platform maturity, automation, CI/CD, observability, and ongoing governance.

Requirements:

• Strong hands-on experience implementing and administering Databricks in an enterprise environment.
• Proven experience designing and implementing Unity Catalog.
• Deep understanding of Databricks account architecture, workspaces, metastores, catalogs, schemas, tables, views, volumes, external locations, storage credentials, and access controls.
• Experience with cloud data platforms such as Azure Databricks and ADLS Gen2, or Databricks on AWS with S3, or Databricks on Google Cloud.
• Strong knowledge of data governance, metadata management, access control, data lineage, auditing, and data classification.
• Experience with Delta Lake, Spark, SQL, Python, notebooks, jobs, workflows, and data engineering pipelines.
• Knowledge of identity and access management, including Entra ID / Azure AD, SCIM provisioning, service principals, groups, and role-based access control.
• Experience defining cluster policies, compute standards, SQL warehouses, cost controls, and operational guardrails.
• Ability to work with infrastructure, security, data governance, and application teams.
• Strong documentation, communication, and stakeholder management skills.

Preferred:

• Experience implementing Databricks using Infrastructure as Code, such as Terraform, Azure DevOps, GitHub Actions, or CI/CD pipelines.
• Experience with enterprise data governance frameworks, data stewardship, data cataloging, privacy, and compliance.
• Experience with Power BI, Microsoft Fabric, Azure Synapse, Azure Data Factory, Purview, or other Microsoft data ecosystem tools.
• Experience designing medallion architecture, lakehouse architecture, or enterprise data platform operating models.

#J-18808-Ljbffr