Senior Python Data Engineer

Daltix is enabling retailers & suppliers to make decisions based on data rather than gut-feeling and for that it’s built up significant experience in how to collect data but also how to transform it in order to support decision making.  

We scrape around 3TB of compressed data per month (20TB uncompressed), if you'd like to learn how this is done and the challenges that comes with that, here's your chance! To this end, we’re looking for a Senior Python Data Engineer, who’ll aid some of the biggest names in the industry in becoming truly data-driven (don’t take our word for it, check our website). 

You will join our data teams who are in charge of standardizing and extracting information from the data we collect, as well as making it accessible for analytics & reporting. Your responsibilities will involve both Data Engineering and Data Analysis skills. 

You will aid us with:

1. Adding new data processing modules to our pipeline so we can standardise data collected from the web.

2. Managing the infrastructure (schedulers, computing frameworks) used for our big scale data processing & reporting.

3. Quality Assurance of our data. We have some tooling in place already, but some more might be necessary. You will likely want to automate some of these checks.

4. Assist with existing ETL pipelines that make our data ready to use by our customers.

5. Enabling our professional services team by providing Python based toolkits to make their jobs easier.

What Daltix offers:

• Private health insurance, a solid laptop (MacBook, Linux-friendly or Windows - it's up to you) and a lot of flexibility!

• The opportunity for you to work only 4 days a week, if that's what you prefer!

• Central based office located near São Sebastião Metro Station. We are a remote working friendly company. At the moment, we are working 100% remotely until the end of 2021 due to the pandemic. Afterwards, we will continue to adopt a hybrid model.

• Your future colleagues will be Nelson Torres (Data Engineer), and Miguel Almeida (Data Scientist). Your team leader will be  Manuel Garrido (Data Architect).

• Work with a modern tech stack including: Python, Docker, Terraform, AWS (S3, Batch), Grafana, Airflow, Snowflake & Looker.

• Best practices for software engineering including mandatory code reviews, unit tests and benchmarks running on every commit, infrastructure-as-code, among others. We're not where we want to be yet, so there's room to add your touch here.

• Squad rotations, allowing you to spend some time per week doing work with another team and learn more about the challenges other colleagues are facing.

Requirements

• At least 5 years of relevant working experience in Data Engineering; we also value knowledge of Data Analysis, however most of the tasks at first will be Data Engineering.

• Must-have tech experience (we use everyday):

  • Python, as 99% of our stack is in Python

  • SQL

  • Git

  • Bash

  • VIM (nah just kidding, it's the best one though)

• Nice-to-have tech experience (we use everyday so it's helpful if you know them too):

  • Pandas + Jupyter notebook

  • Regex

  • CI / CD

  • Docker

  • Cloud experience (AWS preferred)

• The application process involves a technical challenge. Interviews and the challenge will be conducted remotely.

• We communicate exclusively in English, so fluent technical English is mandatory. Portuguese is not required. Most of our data is in Dutch, so knowledge of Dutch is a plus.