Data Engineer - Web Scraper
United Kingdom - Remote
Heni
HENI is an international art services business working with leading artists and estates across publishing, print-making.HENI is looking to increase its data-gathering capability and we are looking for Python Developers (Junior through to Senior) to help us do this. As a Python Developer/Web Scraper the role is split into three main areas:
- Maintenance of existing crawlers
- Setting up new crawlers
- Scraping the NFT/Blockchain market
Tooling
From a tooling perspective, we use Python (Scrapy) to set up the spiders. We have an AWS based infrastructure and data is stored in S3 buckets as well as SQL based databases (MySQL and PostgreSQL).
Requirements
Key Responsibilities:
- Writing and maintaining software for digital data collection for hundreds of websites
- Developing software to allow for scaling of data gathering across thousands of sources and across multiple team members
- Working with and deploying data pipelines (e.g. Scrapy, written in Python), process and cleaning the data, and storing accurately into our database
- Setting up scrapers, fixing any bugs or issues when websites changes
Benefits
- Competitive salary + bonus
- Work in a dynamic and fast-paced environment with new challenges
- Work with a modern tech stack and the latest frameworks
- Have your say – a real chance to influence the tech stack
- Working with a company that uses the most up to date blockchain technology
- Get involved in a variety of projects, see how they develop into polished products and services
- Flexible & fully remote working options – you choose where you work
- Competitive holiday allowance
- Full private healthcare
- Visa + relocation support
- Learning budget for personal development
Need to know information:
- This role can be fully remote, anywhere across the globe.
- We are looking for people who can do this full time (40 hours per week).
- We can offer a competitive salary or hourly rate. It's almost impossible to put a figure on this as different countries have different living costs but we are very competitive.
- If you are based in the UK, we can offer a permanent contract. If you are based overseas, we can offer a freelance contract.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: AWS Blockchain Data pipelines MySQL Pipelines PostgreSQL Python SQL
Perks/benefits: Career development Competitive pay Flex hours Relocation support
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Lead Data Analyst jobs
- Open MLOps Engineer jobs
- Open Data Science Manager jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Manager jobs
- Open Data Engineer II jobs
- Open Power BI Developer jobs
- Open Principal Data Engineer jobs
- Open Sr Data Engineer jobs
- Open Data Analytics Engineer jobs
- Open Business Intelligence Developer jobs
- Open Junior Data Scientist jobs
- Open Data Scientist II jobs
- Open Product Data Analyst jobs
- Open Senior Data Architect jobs
- Open Sr. Data Scientist jobs
- Open Business Data Analyst jobs
- Open Big Data Engineer jobs
- Open Data Analyst Intern jobs
- Open Azure Data Engineer jobs
- Open Manager, Data Engineering jobs
- Open Data Product Manager jobs
- Open Data Quality Analyst jobs
- Open Junior Data Engineer jobs
- Open Principal Data Scientist jobs
- Open Data quality-related jobs
- Open Business Intelligence-related jobs
- Open GCP-related jobs
- Open ML models-related jobs
- Open Data management-related jobs
- Open Java-related jobs
- Open Privacy-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open APIs-related jobs
- Open Deep Learning-related jobs
- Open PyTorch-related jobs
- Open Snowflake-related jobs
- Open TensorFlow-related jobs
- Open Consulting-related jobs
- Open PhD-related jobs
- Open CI/CD-related jobs
- Open NLP-related jobs
- Open Kubernetes-related jobs
- Open Data governance-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open LLMs-related jobs
- Open Databricks-related jobs
- Open Data warehouse-related jobs