Lead Data Engineer

Responsibilities

• Design and develop robust data pipelines and ETL processes for ingesting, transforming, and integrating data from various sources into the data lake.
• Optimize data infrastructure for performance, reliability, and scalability, considering factors such as data volume, velocity, and variety.
• Ensure data quality, integrity, and security across the data lake, implementing data governance practices and monitoring mechanisms.
• Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and develop solutions that meet business needs.
• Champion engineering best practices and drive improvements in data engineering methodologies, processes, and technologies.
• Stay updated with emerging trends and advancements in data engineering, sharing knowledge and insights with the team and the broader organization.
• Effectively communicate technical concepts, solutions, and recommendations to both technical and non-technical stakeholders.

Required Qualifications

• 8+ years of professional experience in data engineering, building scalable data pipelines and ETL processes.
• Strong proficiency in programming languages commonly used in data engineering, such as Java, Scala, or Python.
• In-depth knowledge of distributed systems, data processing frameworks (such as Hadoop, Spark, or Flink), data lake, and cloud computing platforms (e.g., AWS, Google Cloud, Azure).
• Proficient in SQL and relational database technologies (e.g., MySQL, PostgreSQL)
• Strong knowledge of data modeling, and data warehousing concepts.
• Familiarity with data governance, security, access controls, and compliance principles and solutions.
• Demonstrated ability to optimize data pipelines for performance, scalability, and reliability.
• Excellent problem-solving and analytical skills, with a passion for tackling complex data engineering challenges.
• Excellent influencing and communication skills, with the ability to inspire and motivate others toward a shared vision.

Preferred Qualifications

• Experience with real-time data processing and streaming frameworks (e.g., Kafka, AWS Kinesis).
• Knowledge of modern data lakehouse technologies such as Apache Hudi, Delta Lake, or Apache Iceberg
• Experience with  NoSQL databases (e.g., Redis, Cassandra)
• Experience with data visualization tools (e.g., Tableau, Metabase) and data exploration techniques.
• Previous experience leading a team of data engineers or providing technical leadership in data engineering projects.
• Contributions to open-source data engineering projects or active participation in the data engineering community.

Education & Experience

• Bachelor's degree in Computer Science or a related technical field. Advanced degrees are preferred but not required. Equivalent practical experience will also be considered.