Talent.com
Databricks Engineer

Databricks Engineer

Smart IT Frame LLCToronto, ON, Canada
3 days ago
Job type
  • Full-time
Job description

Hi There,

We have a contract role with one of our clients. Please find the below details.

Role : Databricks Engineer

Location : Toronto, Ontario (Hybrid)

Type : Contract

Required Skills :

  • Proficiency in Databricks and Apache Spark.
  • Experience with one or more cloud platforms (Azure Data Lake, AWS S3, GCP BigQuery).
  • Strong programming skills in Python, Scala, or Java.
  • Proficiency in SQL for data querying and manipulation.
  • Familiarity with data orchestration tools like Apache Airflow or Azure Data Factory.
  • Data Expertise :
  • Knowledge of big data technologies (Delta Lake, Hive, Kafka).
  • Understanding of data modeling, warehousing, and database concepts.
  • Soft Skills :
  • Strong analytical and problem-solving skills.
  • Excellent communication and teamwork abilities.
  • Databricks certifications (e.g., Databricks Certified Data Engineer).
  • Experience with machine learning frameworks and Databricks MLflow.
  • Exposure to REST APIs for data integration.
  • Knowledge of CI / CD tools like Jenkins, GitHub Actions, or Azure DevOps.

Roles and responsibilities :

1. Design, develop, and deploy ETL pipelines using Databricks.

  • Implement end-to-end data solutions, including ingestion, transformation, and storage.
  • Utilize Apache Spark within Databricks for large-scale data processing.

    • 2. Platform Expertise :

  • Develop solutions on Databricks integrated with cloud platforms (Azure, AWS, or GCP).
  • Optimize Databricks clusters and workflows for performance and cost efficiency.

    • 3. Collaboration :

  • Work closely with data scientists, analysts, and stakeholders to understand business requirements.
  • Collaborate with DevOps teams for CI / CD pipeline integration and automation.

    • 4. Data Governance & Security :

  • Implement and maintain data security and compliance measures.
  • Ensure data quality and reliability using robust validation techniques.

    • 5. Troubleshooting & Support :

  • Monitor, troubleshoot, and resolve issues in Databricks workflows and pipelines.
  • Stay updated on platform upgrades, best practices, and new features.