Lead Data Engineer

ClearScore is a workplace like no other. We’ve spent the past five years disrupting an entire industry and building a user base of millions. At the heart of this success is our culture, where we work hard, relish change and treat each other with respect—helping everyone to reach their potential and deliver results that make a difference.

We’re on a mission to positively impact the lives of our users. Our product puts their needs at the centre of our thinking. Through cutting-edge tech, insightful analytics and beautiful design we want to help our users build their financial confidence and make better financial decisions.

We want our people to perform at their best, so we trust everyone to work in a way that suits them—focusing on output, not time spent at a screen. We have an inclusive culture where everyone is encouraged to look after their own wellbeing whilst growing and developing their career.

For more information on our tech stack check out our Tech Radar 2020, how we work is summarised in our Engineering Principles and we have many other Tech Blogs on Medium.

We’re looking for an Engineering Manager to help the Clearscore Data team build a world class data processing platform. You’ll be working on solving data challenges across a range of systems, from real-time event streams powering our CRM and other data integrations to traditional batch workloads underpinning our warehouse and machine learning models.

The systems you’ll be working on process internet scale event data generated by our customer base of over 14 million users and are the backbone for all data use cases at Clearscore.

The team and our existing stack

We’re an agile team of data engineers and backend developers responsible for building and operating a data processing platform that handles tracking events, credit report data and other streams from all ClearScore apps across the UK, Australia and South Africa:

• Near-real time ingestion pipelines handling >100M events per day, built using Kinesis, Kafka and Spark-streaming and written in Scala
• Batch pipelines written in Spark or SQL, with scheduling and dependencies defined in Airflow
• S3 data lake holding >60TB of data, underpinning our Redshift Spectrum warehouse
• Modern, AWS based infrastructure using k8s and automated CI/CD with Jenkins and Terraform, making the team fully autonomous
• Design and build a scalable data platform to allow business users and applications to access data
• Enable a world class personalisation platform, allowing our data scientists to deploy models that will improve user experience throughout the app
• Create north star strategy for data
• Create data as a service to allow us and our partners to gain value from the data we hold in a seamless and frictionless way
• Tackle high impact problems around data lineage, discoverability, and accessibility to enable the business to build upon now and into the future
• Implement a team structure and dynamic that scales to accommodate future growth
• Create high performing teams where metrics are used to experiment and improve ways of working
• Several years' experience building high performance streaming and batch data processing systems
• A high degree of proficiency with Python, Java or Scala
• Solid knowledge of distributed processing with tools like Kafka and Spark
• Good understanding of AWS data stack (s3, MSK, EMR, Glue, Redshift etc)
• Deep knowledge of any elastic warehouse technology (snowflake, redhshift, bigquery etc)
• Line management experience of teams of 8-10 engineers
• Good understanding of different agile methodologies and know how to create high performing teams
• Experience leading initiatives outside of your team that impact cross business functions
• Cloud certifications (AWS, Google, Azure etc)
• Big data tool certifications (snowflake, data bricks etc)
• Can demonstrate times you have used metrics/data to gauge success
• Can demonstrate examples of non-technical and technical solutions you have implemented to solve issues you have faced
• 25 paid holidays and a “duvet day” on your birthday
• Private health and dental cover
• GP office visits
• Life assurance scheme
• Up to 6% matched pension
• Generous maternity and paternity plans
• Generous training allowance
• Leadership-led training
• Regular wellbeing events
• In-house psychotherapist
• Financial Advice through Hatch
• Access to Perkbox
• Dog-friendly office
• Daily breakfast and free snacks
• Free sports and social clubs
• Fast progression
• Physical and mental health support through BUPA
• No clock-watching culture
• Culture and inclusion representatives
• Brown bags with guest speakers

The challenge

Over the last couple of years, we managed to build a complex, end to end data platform that powers many use cases across Clearscore. As the company has continued to grow, we are now looking to make it more robust, scalable and efficient. To achieve we will need to solve old and new challenges including:

Requirements

A plus if you have

Benefits Work with latest technology with minimal legacy debt