Azure Databricks Engineer

Job Title: Senior Data Engineer

Location: Remote

Experience Level: 5-7 Years

Job Description:

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, particularly with Azure Databricks, Pyspark, Medallion Architecture, and Data Lakes. If you are passionate about working with large datasets and driving data solutions, we want to hear from you.

Responsibilities:

• Design, develop, and optimize data pipelines using Azure Databricks and Pyspark.
• Implement and maintain Medallion Architecture to streamline data processing workflows.
• Manage and organize large datasets in Data Lakes, ensuring data quality and integrity.
• Write complex SQL queries to extract, transform, and load data from various sources.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions.
• Monitor and troubleshoot data pipeline performance and resolve issues as they arise.
• Stay up-to-date with the latest advancements in data engineering and propose innovative solutions to improve data processes.

Requirements:

• 5-7 years of hands-on experience in data engineering or a related field.
• Proficiency in Azure Databricks and Pyspark.
• Strong understanding of Medallion Architecture and its implementation.
• Experience working with Data Lakes and managing large datasets.
• Advanced SQL skills with the ability to write and optimize complex queries.
• Knowledge of Azure services and cloud-based data solutions.
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration abilities.