Data Engineer (Python/Spark/AWS) - Remote
Chicago, IL
Nielsen
A global leader in audience insights, data and analytics, Nielsen shapes the future of media with accurate measurement of what people listen to and watch.Data Engineer (Python/Spark/AWS) - Remote - 85785Technology and Engineering - USA Chicago, Illinois
The Global Outcomes with Nielsen team is looking for a Senior Data Engineer to help us bring data-intensive products to market, maintain existing products for our clients, and work closely with our data science teams in a cloud-native, Python and Spark-heavy big data stack.
A typical day at this role includes attending a standup with data engineers, data scientists, and our product owner, talking about various data sources we’re integrating into our machine learning pipelines with upstream Nielsen teams, guiding data scientists in how to access the data, and turning their analyses into production-ready Spark code that runs using Airflow.
Role Details:Work with other data engineers, data scientists, architects, and product owners on an agile scrum team that delivers products to production.Gather, analyze and convert business requirements into AWS (Amazon Web Services) cloud-based solutions.Design and build systems that load and transform a large volume of structured and semi-structured data. When we say “big data” we don’t mean a few gigabytes – we work with multi-terabyte datasets on a daily basis.Build and test cloud-based data pipelines and applications (primarily in Python and Apache Spark + SQL) for new and existing backend systems.Write reusable, well-tested code and components (e.g. RESTful APIs, Python packages, etc.) that can be used by multiple project teams.Assist in troubleshooting and debugging of ETL code and resolving data integrity issues alongside our data scientists and client-facing customer success teams.Work in a serverless environment. We don’t maintain VMs nor do we manually deploy infrastructure. Automation and scalability is critical.Write code with performance, maintainability, scalability, and reliability in mind.Our tech stack: Python, Apache Spark, SQL, Apache Airflow, Hive, AWS Glue, AWS Athena, AWS EC2, AWS S3, AWS CodeBuild, AWS CloudFormation, YARN, Git, RESTful Microservices, Kubernetes (k8s).
Role Qualifications:Master’s degree in computer science, engineering, or a related field with an information technology focus (foreign equivalent degree acceptable) plus 3 years of experience in software design and development.ORBachelor’s degree in computer science, engineering, or a related field with an information technology focus (foreign equivalent degree acceptable) plus 5 years of experience in software design and development
3 years of experience with:delivering end-to-end applications and pipelines (including architecting open source-based ETL pipelines and designing, building and implementing big data solutions).2 years of experience with:AWS, Azure, or Google Cloud Platform, preferably in a serverless tech stack.Designing and developing Apache Spark-based applications using Python (PySpark) or Scala and Spark SQL.Comfort with the Linux command line, Git, Agile Scrum and at least one data orchestration tool e.g. Apache Airflow, Luigi, Azkaban, AWS Data Pipeline, Oozie, etc..
#LI-TN1ABOUT NIELSEN
As the arbiter of truth, Nielsen Global Media fuels the media industry with unbiased, reliable data about what people watch and listen to. To discover what’s true, we measure across all channels and platforms—from podcasts to streaming TV to social media. And when companies and advertisers are armed with the truth, they have a deeper understanding of their audiences and can accelerate growth. Do you want to move the industry forward with Nielsen? Our people are the driving force. Your thoughts, ideas and expertise can propel us forward. Whether you have fresh thinking around maximizing a new technology or you see a gap in the market, we are here to listen and take action. Our team is made strong by a diversity of thoughts, experiences, skills, and backgrounds. You’ll enjoy working with smart, fun, curious colleagues, who are passionate about their work. Come be part of a team that motivates you to do your best work! Nielsen is committed to hiring and retaining a diverse workforce. We are proud to be an Equal Opportunity/Affirmative Action-Employer, making decisions without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability status, age, marital status, protected veteran status or any other protected class.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Agile Airflow APIs Athena AWS Azkaban Azure Big Data Computer Science Data pipelines EC2 Engineering ETL GCP Git Google Cloud Kubernetes Linux Machine Learning Microservices Oozie Open Source Pipelines PySpark Python Scala Scrum Spark SQL Streaming
Perks/benefits: Career development
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open MLOps Engineer jobs
- Open Lead Data Analyst jobs
- Open Data Science Manager jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Manager jobs
- Open Data Engineer II jobs
- Open Sr Data Engineer jobs
- Open Principal Data Engineer jobs
- Open Data Analytics Engineer jobs
- Open Power BI Developer jobs
- Open Business Intelligence Developer jobs
- Open Junior Data Scientist jobs
- Open Data Scientist II jobs
- Open Senior Data Architect jobs
- Open Product Data Analyst jobs
- Open Sr. Data Scientist jobs
- Open Business Data Analyst jobs
- Open Manager, Data Engineering jobs
- Open Big Data Engineer jobs
- Open Data Analyst Intern jobs
- Open Data Quality Analyst jobs
- Open Data Product Manager jobs
- Open Azure Data Engineer jobs
- Open ETL Developer jobs
- Open Junior Data Engineer jobs
- Open Data quality-related jobs
- Open Business Intelligence-related jobs
- Open ML models-related jobs
- Open GCP-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open Java-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open APIs-related jobs
- Open Deep Learning-related jobs
- Open PyTorch-related jobs
- Open Consulting-related jobs
- Open Snowflake-related jobs
- Open TensorFlow-related jobs
- Open PhD-related jobs
- Open CI/CD-related jobs
- Open NLP-related jobs
- Open Data governance-related jobs
- Open Kubernetes-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open Databricks-related jobs
- Open DevOps-related jobs
- Open LLMs-related jobs