Senior Data Engineer

ClearScore is a workplace like no other. We’ve spent the past five years disrupting an entire industry and building a user base of millions. At the heart of this success is our culture, where we work hard, relish change and treat each other with respect—helping everyone to reach their potential and deliver results that make a difference.

We’re on a mission to positively impact the lives of our users. Our product puts their needs at the centre of our thinking. Through cutting-edge tech, insightful analytics and beautiful design we want to help our users build their financial confidence and make better financial decisions.

We want our people to perform at their best, so we trust everyone to work in a way that suits them—focusing on output, not time spent at a screen. We have an inclusive culture where everyone is encouraged to look after their own wellbeing whilst growing and developing their career.

For more information on our tech stack check out our Tech Radar 2020, how we work is summarised in our Engineering Principles and we have many other Tech Blogs on Medium.

We’re looking for a Senior Data Engineer to help the Clearscore Data team build a world class data processing platform. You’ll be working on solving data challenges across a range of systems, from real-time event streams powering our CRM and other data integrations to traditional batch workloads underpinning our warehouse and machine learning models.

The systems you’ll be working on process internet scale event data generated by our customer base of over 14 million users and are the backbone for all data use cases at Clearscore.

The team and our existing stack

We’re an agile team of data engineers and backend developers responsible for building and operating a data processing platform that handles tracking events, credit report data and other streams from all ClearScore apps across the UK, Australia and South Africa:

• Near-real time ingestion pipelines handling >100M events per day, built using Kinesis, Kafka and Spark-streaming and written in Scala
• Batch pipelines written in Spark or SQL, with scheduling and dependencies defined in Airflow
• S3 data lake holding >60TB of data, underpinning our Redshift Spectrum warehouse
• Modern, AWS based infrastructure using k8s and automated CI/CD with Jenkins and Terraform, making the team fully autonomous

The challenge

Over the last couple of years, we managed to build a complex, end to end data platform that powers many use cases across Clearscore. As the company continues to grow, we’re looking to make it more robust, scalable and efficient, to make sure it scales with the business:

• Simplify our ingestion and integration pipelines using modern stream processing tools to reduce cost and latency
• Make our batch processing more robust, performant and easier to maintain by making sure jobs are fully idempotent and optimised
• Improve observability and monitoring to ease the maintenance workload
• Start building a modelled data warehouse to make life easier for analysts and data scientists across the business
• Build a world class personalisation platform, allowing our data scientists to deploy models that will improve user experience throughout the app

Requirements

• Several years experience building high performance streaming and batch data processing systems
• A high degree of proficiency with Python, Java or Scala
• Solid knowledge of distributed processing with tools like Kafka and Spark
• Good understanding of AWS data stack (s3, MSK, EMR, Glue, Redshift)
• Experience working in an agile environment, using DataOps principles

Benefits

• 25 paid holidays and a “duvet day” on your birthday
• Private health and dental cover
• GP office visits
• Life assurance scheme
• Up to 6% matched pension
• Generous maternity and paternity plans
• Generous training allowance
• Leadership-led training
• Regular wellbeing events
• In-house psychotherapist
• Financial Advice through Hatch
• Access to Perkbox
• Dog-friendly office
• Daily breakfast and free snacks
• Free sports and social clubs
• Fast progression
• Physical and mental health support through BUPA
• No clock-watching culture
• Culture and inclusion representatives
• Brown bags with guest speakers

Inclusion Policy

We are always looking for talented individuals to join ClearScore. We are an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for our people. Please see our People Policy Notice at https://www.clearscore.com/people-notice.