Lead Data Engineer

About Confused.com

Confused.com is the UK’s first comparison platform for car insurance. We’ve been helping customers since 2002 by empowering them to make better decisions around insurance and financial services. Our mission is simple: take away the confusion when comparing financial products and services to help you save time and money.

We’re part of RVU - a global group of online brands (including Uswitch and Money.co.uk) that work to empower consumers. We do this by helping people compare home services, insurance and other financial products.

About the role

We have an exciting opportunity for an experienced Lead Data Engineer working in an environment using cutting edge technologies alongside a team of like-minded data enthusiasts.

Working in this fast-paced, agile environment, you will collaborate with a team of other data and market experts which offers a continuous delivery release-driven environment focused around value.

We have lots of exciting projects which will give you opportunities to develop and help develop others whilst leveraging the latest technologies and delivering business value.

What you’ll be doing:

• Working closely with other departments to integrate and implement data engineering solutions giving you opportunities to expand your knowledge and explore new capabilities
• Identifying, designing, and implementing internal process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalability etc
• Fostering a self-service data culture; working closely with other data engineers, architects, and technologists to share knowledge, evaluate opportunities and proactively recommend improvements in architecture, pipelines, data models, tools and processes
• Translating data into valuable insights that inform decisions, which involves teams in analytics and synthesis to increase consensus and challenge assumptions
• Maintaining and creating continuous delivery pipelines in Azure DevOps

Requirements

Essential skills:

The ideal candidate will have a proven track record in implementing data ingestion and transformation pipelines with deep technical skills in a variety of technologies including:

- Python

- Databricks (or Apache Spark)

- SQL

- Infrastructure-as-code

● Knowledge of data warehousing concepts, including data warehouse technical architectures, infrastructure components, ETL/ELT and reporting/analytic tools

● Ability to understand detailed technical requirements and explain technical concepts to both technical and non-technical stakeholders alike

Nice to have

● Demonstrable understanding of how to expose data from systems (eg. through APIs), link data from multiple systems and deliver streaming services.

● Experience working with structured and unstructured data

● Experience with code versioning systems (e.g. git, VSTS) and CI/CD tools such as MS Build
● Strong experience designing, implementing and maintaining robust data pipelines and transformation processes using Azure Data Factory

You don’t need to tick off everything on this list - so don’t let that hold you back from applying. There are plenty of opportunities to learn with us!

Our commitment to you:

At Confused.com we believe that we can be the change we wish to see in the world. We hold ourselves accountable to being open and inclusive teammates and community members. We embrace our differences and are committed to creating an inclusive environment that reflects the world we live in.

Benefits

We want to give you a great work environment. We want to contribute to both your personal and professional development, and give you great benefits to make your time at Confused.com even more enjoyable. Some of these benefits include:

• A competitive salary and bonus package
• Employer matching pension up to 7.5%
• A hybrid approach of in-office and remote working, and a “Work from Home” budget to help contribute towards a great work environment at home
• Annual leave of 25 days (excluding bank holidays) which will increase with length of service (up to a maximum of 30 days) and additional leave options.
• Excellent maternity, paternity and adoption leave policy, for those key moments in your life
• Employee Assistance Programme (EAP), Simply Health Scheme and access to the Calm app.
• Regular events - company-wide events with insightful external speakers as we want to make sure our colleagues continue to feel connected.
• A healthy learning and training budget, as well as the chance to go to conferences around the world every year
• Private medical scheme

If you would like to know more, please get in touch by emailing recruitment@confused.com

Confused.com are proud to have been officially recognised as a 2022 UK’s Best Workplace™ and as a 2022 UK’s Best Workplace™ for Wellbeing in the Large size category!