Lead AWS Data Engineer

Job Description

About you:

As a lead data engineer, you will design and maintain data platform road maps and data structures that support business and technology objectives.  Naturally inquisitive and open to the deep exploration of underlying data, finding actionable insights, and working with functional competencies to drive identified actions.  You also enjoy working both freely and as part of a team and have the confidence to influence and communicate with stakeholders at all levels, and to work in a fast-paced complex environment with conflicting priorities.

About the role:

Reporting into the delivery leader, you will deliver consumable, contemporary, and immediate data content to support and drive business decisions. The key focus of the role is to deliver a custom solution to support various business critical requirements. You will be involved in all aspects of data engineering from delivery planning, estimating and analysis, all the way through to data architecture and pipeline design, delivery, and production implementation.  From the beginning you will be involved in the design and implementation of complex data solutions ranging from batch to streaming and event-driven architectures, across cloud, on-premise, and hybrid client technology landscapes.

 Brief Description of Role:

We are looking for 5+ years of experience in data engineering in a customer or business facing capacity and experience in the following:

• Ability to understand and articulate requirements to technical and non-technical audiences
• Stakeholder management and communication skills, including prioritizing, problem solving and interpersonal relationship building
• Strong experience in SDLC delivery, including waterfall, hybrid, and Agile methodologies
• Experience of implementing and delivering data solutions and pipelines on AWS Cloud Platform.
• A strong understanding of data modelling, data structures, databases, and ETL processes
• An in-depth understanding of large-scale data sets, including both structured and unstructured data
• Knowledge and experience of delivering CI/CD and DevOps capabilities in a data environment
• Develop new inbound data source ingestions required within the multi-tiered data platform to support analytics and marketing automation solutions
• Supports data pipelines – Builds the required dimensions, rules, segments, and aggregates
• Support all database operations: performance monitoring, pipeline ingestion, maintenance, etc.
• Monitor platform health - data loads, extracts, failures, performance tuning
• Create/modify data structures/pipelines
• Leveraging capabilities of Databricks Lakehouse functionality as needed to build Common/Conformed layers within the data lake
• Application of software best practices into consideration. E.g. infrastructure cost, security, scalability, feature extensibility, ease of maintenance, etc.
• Develop, document, and test software and environment setup to ensure that the outcome meets the needs of end-users and achieves business goals

Qualifications

The following skills are required:

• Tech Stack: AWS pipeline, Glue, Databricks, Python, SQL, Spark, etc.
• Building the Data Lake using AWS technologies like S3, EKS, ECS, AWS Glue, AWS KMS, EMR
• Extensive experience in ETL and audience segmentation
• Developing sustainable, scalable, and adaptable data pipelines
• Attention to detail in design, documentation, and test coverage of delivered tasks
• Strong written and verbal communication skills, team player
• Strong analytical and leadership skills
• In addition, the candidate should have strong business acumen, interpersonal skills, and communication skills, yet also be able to work independently.
• At least 4 years of experience with designing and developing Data Pipelines for Data Ingestion or Transformation using AWS technologies
• At least 3 years of experience in the following Big Data frameworks: File Format (Parquet, etc.), Resource Management, Distributed Processing
• At least 4 years of experience developing applications with Monitoring, Build Tools, Version Control, Unit Test, TDD, Change Management to support DevOps
• At least 4 years of experience with Spark programming (PySpark or Scala)
• At least 4 years of experience with Databricks implementations
• Familiarity with the concepts of “delta lake” and “lakehouse” technologies
• At least 3 years of experience working with Streaming using Spark or Flink or Kafka

The following skills are nice to have, and expertise is not required:

• Adobe (Campaign, Audience Manager, Analytics)
• MLFlow
• Microsoft Power BI
• SAP Business Objects

Additional Information

When you’re one of us, you get to run with the best. For decades, we’ve been helping marketers from the world’s top brands personalize experiences for millions of people with our cutting-edge technology, solutions and services. Epsilon’s best-in-class identity gives brands a clear, privacy-safe view of their customers, which they can use across our suite of digital media, messaging and loyalty solutions. We process 400+ billion consumer actions each day and hold many patents of proprietary technology, including real-time modeling languages and consumer privacy advancements. Thanks to the work of every employee, Epsilon has been consistently recognized as industry-leading by Forrester, Adweek and the MRC. Positioned at the core of Publicis Groupe, Epsilon is a global company with more than 8,000 employees around the world. Check out a few of these resources to learn more about what makes Epsilon so EPIC:

• Culture: https://www.epsilon.com/us/about-us/our-culture-epsilon
• DE&I: https://www.epsilon.com/us/about-us/diversity-equity-inclusion
• CSR: https://www.epsilon.com/us/about-us/corporate-social-responsibility
• Life at Epsilon: https://www.epsilon.com/us/about-us/epic-blog

Great People Deserve Great Benefits

We know that we have some of the brightest and most talented associates in the world, and we believe in rewarding them accordingly. If you work here, expect competitive pay, comprehensive health coverage, and endless opportunities to advance your career.

Epsilon is an Equal Opportunity Employer.  Epsilon’s policy is not to discriminate against any applicant or employee based on actual or perceived race, age, sex or gender (including pregnancy), marital status, national origin, ancestry, citizenship status, mental or physical disability, religion, creed, color, sexual orientation, gender identity or expression (including transgender status), veteran status, genetic information, or any other characteristic protected by applicable federal, state or local law. Epsilon also prohibits harassment of applicants and employees based on any of these protected categories.

Epsilon will provide accommodations to applicants needing accommodations to complete the application process.

#LI-SJ1

REF186909L