Principal Data Engineer
United States (Remote)
Swiftly
Elevate your retail business with our innovative Retail Technology and Retail Media Network. Offering specialized solutions for both retailers and advertisers, Swiftly provides retail media solutions and drives measurable growth for both...Position Summary: Swiftly is searching for a Principal Data Engineer that will play a critical role in designing, implementing, and managing the data infrastructure of our organization. The ideal candidate will have expertise that will contribute to the availability, usability, and integrity of data, enabling data-driven decision-making and business success.
Responsibilities Include:
- Design, build, deploy, and maintain ETL (Extract, Transform, Load) processes using Azure hosted technologies
- Orchestrate data workflows to integrate structured and unstructured data from multiple sources
- Manage customer import and export processes that involve Secure File Transfer Protocol (SFTP), Azure Storage and AWS S3 for data exchange
- Ensure data security and compliance while transferring data between systems
- Support our inclusive and supportive engineering culture, Mentor engineering team
- Advocate for Agile development methodologies and participate in the software development lifecycle (SDLC) activities
- Implement and maintain security measures to protect company assets and customer data
- Develop contingency plans to address potential issues
- Collaborate with data scientists, software engineers, and other stakeholders to create efficient data pipelines using Databricks
- Optimize data processing at scale using Databricks’ capabilities while managing costs
- Author REST servers and client to enhance data and provide admin portal functionality
- Develop models and data sets to enhance personalized services for retail grocery shoppers
- Demonstrate how to leverage data to improve customer experiences through effective targeting
- Other related duties as assigned
Required Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or a related field
- 10+ years of Data Engineering experience
- 5+ years in a small team, startup environment
- Strong knowledge of Python
- Foundational understanding of Azure, Eventhubs, storage accounts and infrastructure considerations
- Experience with SQL, message queuing, stream processing, and advertising platforms
- Deep experience with data pipelines (ETL/ELT, Spark, dbt, Airflow, Databricks, etc.)
- Deep experience with OLAP database design, build, run (Star/ Snowflake, Fact / Dim, Snowflake, etc.)
- Deep experience with distributed application design, build, run (api, queues, idempotent, encapsulated, etc.)
- Experienced with cloud infrastructure (infra as code, cdk, Terraform, Kubernetes, containers & images
- Proven experience in a leadership role within engineering, with a track record of successful project delivery
- Excellent communication, and interpersonal skills
Preferred Qualifications:
- 10+ years in a customer facing, B2B or SaaS data engineering
- Large corporate experience (in addition to startup experience)
- Experience working in a distributed (remote first) environment
- Deep experience using Databricks
- Experience with Postgres
- Deep experience with OLTP database design, build, run (SQL, normalization, query optimization, Databricks, etc.)
Working For SwiftlyWe are working on a set of problems that require the best in the industry to get right. Our employees are the pillars of the future of the company and they are treated and compensated as such. We're a growing team of experienced industry professionals building an organization that can solve tough problems and values a collaborative environment.
Every Swiftly employee: -Has demonstrated the ability to work collaboratively in an ambiguous, fast-paced environment -Takes ownership of their domain from the ground up, from inception through deployment to customers -Leaves their ego at the door and ensures the best idea leaves the room -Is always experimenting with new technologies and learning new skillsets
If you've ever wanted to work on a project that blends a gorgeous consumer experience, sophisticated logistics, and scale that rivals the largest technology players, Swiftly is the place for you.
Equal Opportunity Employer Swiftly is an Equal Employment Opportunity and Affirmative Action Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status. Swiftly will make reasonable accommodations in compliance with the Americans with Disabilities Act of 1990. Reasonable accommodations are available for qualified individuals with disabilities throughout the subsequent application process.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Agile Airflow APIs AWS Azure Computer Science Databricks Data pipelines dbt ELT Engineering ETL Kubernetes OLAP Pipelines PostgreSQL Python SDLC Security Snowflake Spark SQL Terraform Unstructured data
Perks/benefits: Startup environment Team events
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Data Engineer II jobs
- Open Business Intelligence Engineer jobs
- Open Lead Data Analyst jobs
- Open Power BI Developer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Marketing Data Analyst jobs
- Open Data Science Manager jobs
- Open Junior Data Scientist jobs
- Open MLOps Engineer jobs
- Open Business Data Analyst jobs
- Open Data Scientist II jobs
- Open Product Data Analyst jobs
- Open Business Intelligence Developer jobs
- Open Data Analytics Engineer jobs
- Open Sr Data Engineer jobs
- Open Data Analyst Intern jobs
- Open Senior Data Architect jobs
- Open Principal Data Scientist jobs
- Open Sr. Data Scientist jobs
- Open Junior Data Engineer jobs
- Open Big Data Engineer jobs
- Open Data Quality Analyst jobs
- Open Manager, Data Engineering jobs
- Open Azure Data Engineer jobs
- Open Research Scientist jobs
- Open GCP-related jobs
- Open Java-related jobs
- Open Data quality-related jobs
- Open ML models-related jobs
- Open Business Intelligence-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open PhD-related jobs
- Open Deep Learning-related jobs
- Open Data visualization-related jobs
- Open NLP-related jobs
- Open Finance-related jobs
- Open PyTorch-related jobs
- Open TensorFlow-related jobs
- Open APIs-related jobs
- Open LLMs-related jobs
- Open Generative AI-related jobs
- Open Consulting-related jobs
- Open CI/CD-related jobs
- Open Snowflake-related jobs
- Open Kubernetes-related jobs
- Open Hadoop-related jobs
- Open Data governance-related jobs
- Open Databricks-related jobs
- Open Data warehouse-related jobs