Data Engineering Lead

Data engineering at Paystack focuses on building platforms for managing data at scale. This involves multiple stages: data ingestion, processing, storage and egress. Data engineers are also responsible for creating and maintaining the infrastructure a data platform runs on. A lead data engineer must be able to handle these responsibilities alongside managing a team of engineers and driving the technical decision making processes the data team are involved in.

Role and responsibilities

A lead data engineer must be capable of operating across a diverse tech stack. They are expected to be adaptable and unafraid of the unknown. The role calls for someone with a good background in software development, data analytics and distributed systems. Furthermore, the lead needs to provide their team with a good working environment and be able to manage stakeholder expectations about their team’s abilities.

A lead data engineer's responsibilities may include:

• Architectural design of a data platform
• Data pipeline maintenance/testing
• Administration of a large cloud data warehouse
• Creation and maintenance of infrastructure-as-code components
• Monitoring system performance and addressing faults in production systems with an on-call rotation
• Development of in-house software
• Regular check-ins with their direct reports
• Development of team rituals and administration

Requirements and skills

Technical skills

• A software engineering background
• Experience with test driven development
• Fluency in a modern programming language, including but not limited to: Java, Scala, Python, C/C++ and Golang
• An understanding of data modelling
• Experience in running ETL pipelines
• Expertise in data analysis
• Knowledge of BI tools
• Event streaming/message broker systems
• Change data capture
• Experience with kubernetes
• Familiarity with a cloud environment (AWS, Azure, GCP)
• Familiarity with MongoDB, Postgresql and mysql

Tools

• Kubernetes
• Kafka Connect
• Debezium
• Prometheus
• AWS Redshift
• dbt
• S3
• CI/CD pipelines
• Workflow orchestration tools (Airflow and Dagster specifically)

Soft skills

• Communication skills
  • Be able to communicate with different stakeholders (data analysts, machine learning engineers, CTOs, and developers)
  • Be able to work with other teams or business units to gather requirements and define the scope of a project
  • They must understand the underlying business problems that they are trying to address and articulate how their solution can help
• Collaboration
  • Be able to work closely with team mates in a remote setting
  • Be able to deal with difficult problems and own their own mistakes

Paystack is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind. We’re committed to providing employees with a work environment that is progressive and open-minded. Our employment philosophy is to hire the best people and empower them to do the best work of their lives. Employment decisions are based on business needs and individual merit without regard to race, color, religion, ethnicity, sexual orientation, nationality, marital status, gender or age.