Senior Data Engineer(Databricks) - Toronto
Toronto, Ontario, Canada
Prodigy Labs
Accelerating Digital Transformation Prodigy Labs provides a comprehensive range of expertise, from advisory services through to final delivery. We apply emerging technologies to enterprise client operations to create real, sustained value in...Data Engineer with Databricks & Spark Experience
We are looking for a data engineer who has strong experience in building scalable and reliable data pipelines using Databricks and Spark. You will be working with various data sources and formats, and transforming them into valuable insights for our business.
Responsibilities
- Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed
- Optimize data pipelines for performance, scalability, and reliability
- Ensure data quality and integrity throughout the data lifecycle
- Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs
- Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations
- Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management
- Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community
- Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies
Please Note: This is a hybrid position and will require at least 2 days in the office per week. Successful candidates will need to complete a background check.
Requirements
Qualifications
- Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent work experience
- 5+ years of experience in data engineering with Databricks and Spark
- Proficient in SQL and Python, and familiar with Java or Scala
- Experience with cloud platforms, such as Azure or AWS
- Experience with big data technologies, such as Kafka, Hadoop, Hive, etc.
- Experience with data warehouse and data lake concepts and architectures
- Experience with data integration and ETL tools, such as Azure Data Factory or Talend
- Experience with data visualization and reporting tools, such as Power BI or Tableau
- Strong analytical and problem-solving skills
- Excellent communication and teamwork skills
Benefits
Benefits
- We are a collective group of people and collaboration is key to our process
- We don’t work for our clients, we work with them
- A Flexible Hybrid Working Environment
- Easily accessible downtown location
- Competitive compensation commensurate with experience
- Everyone brings something valuable to the table in our supportive, challenging, and collaborative, diverse work environment
All Employees Can Participate in:
- Company paid health benefits: 100% medical, dental and vision coverage
- Corporate-discounted Gym Membership through GoodLife
- Company discount program including Travel, Shopping, Attractions, Wellness, & Sporting events, just to name a few
- Access to an Employee & Family Assistance program (EAP)
- Employee Referral Program
- Employee Opportunity Program
- Professional Development Program
- Philanthropic Events
- Social Events
Inclusion & Diversity
We foster an inclusive and diverse workforce, believing our strength stems from our individual differences. Our employees, partners, and clients continuously benefit from the innovation and creativity grounded in these values. We strive to be a company that attracts a diverse group of highly skilled people who know that their contributions will be valued and that they will be heard. We are committed to building a corporate culture with people who are excited to join our team, do their best work, and grow with us!
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Architecture AWS Azure Big Data Computer Science Data analysis Databricks Data governance Data pipelines Data quality Data visualization Data warehouse Engineering ETL Hadoop Java Kafka Pipelines Power BI Python Scala Security Spark SQL Tableau Talend
Perks/benefits: Career development Competitive pay Fitness / gym Flex hours Health care Team events
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Data Engineer II jobs
- Open Business Intelligence Developer jobs
- Open Data Science Manager jobs
- Open Principal Data Scientist jobs
- Open Business Data Analyst jobs
- Open BI Analyst jobs
- Open Data Scientist II jobs
- Open Business Intelligence Engineer jobs
- Open Sr Data Engineer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Lead Data Analyst jobs
- Open Data Science Intern jobs
- Open Sr. Data Scientist jobs
- Open Junior Data Scientist jobs
- Open MLOps Engineer jobs
- Open Software Engineer, Machine Learning jobs
- Open Data Analytics Engineer jobs
- Open Marketing Data Analyst jobs
- Open Azure Data Engineer jobs
- Open Manager, Data Engineering jobs
- Open Data Engineer III jobs
- Open Junior Data Engineer jobs
- Open Data Analyst II jobs
- Open Data Quality Analyst jobs
- Open ETL Developer jobs
- Open Data management-related jobs
- Open GCP-related jobs
- Open Excel-related jobs
- Open Privacy-related jobs
- Open Data pipelines-related jobs
- Open ML models-related jobs
- Open PhD-related jobs
- Open APIs-related jobs
- Open PyTorch-related jobs
- Open Finance-related jobs
- Open LLMs-related jobs
- Open Data visualization-related jobs
- Open Business Intelligence-related jobs
- Open TensorFlow-related jobs
- Open Consulting-related jobs
- Open Deep Learning-related jobs
- Open Generative AI-related jobs
- Open Data governance-related jobs
- Open NLP-related jobs
- Open CI/CD-related jobs
- Open DevOps-related jobs
- Open Kubernetes-related jobs
- Open Snowflake-related jobs
- Open Git-related jobs
- Open Hadoop-related jobs