Senior Data Engineer - Python/AWS/DynamoDB
India
3Pillar Global
3Pillar builds breakthrough software products that power digital businesses. Discover product development services.We are 3PILLAR GLOBALWe build breakthrough software products that power digital businesses. We are an innovative product development partner whose solutions drive rapid revenue, market share, and customer growth for industry leaders in Software and SaaS, Media and Publishing, Information Services, and Retail. Our key differentiator is our Product Mindset. Our development teams focus on building for outcomes and all of our team members around the globe are trained on the Product Mindset’s core values – Minimize Time to Value, Solve For Need, and Excel at Change. Our teams apply this mindset to build digital products that are customer-facing and revenue-generating. Our business-minded approach to agile development ensures that we align to client goals from the earliest conceptual stages through market launch and beyond.
In 2024, 3Pillar Global India was named as a “Great Place to Work” for the seventh year in a row based on how our employees feel about our company, collaborative culture, and work/life balance - come join our growing team
Job Description: As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.If your passion for data analytics solutions that make a real-world impact, consider this your pass to the captivating world of Data Science and Engineering!
Responsibilities
- Understanding the business requirements
- Writing code and implementing the proposed solutions
- Creating data pipelines, versioning and change management
- Manage the complexity inherent in versioned data pipelines
- Develop ETL/ELT processes to help extract and manipulate data from multiple sources.
- Design, build and maintain batch or real-time data pipelines in production.
- Automate data workflows such as data ingestion, aggregation, and ETL processing.
- Logging and instrumentation of pipelines and services.
- Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
- Train and mentor junior engineers.
Requirements
- Demonstrated expertise with a minimum of 5+ years of experience as data engineer or similar role
- Advanced SQL skills and experience with relational databases and database design.
- Strong expertise in DynamoDB, including data modeling, querying, and optimization.
- Experience working with cloud Data Warehouse solutions e.g. Snowflake (or Redshift, BigQuery, Azure Synapse, etc.).
- Strong Python skills with hands-on experience on Pandas, NumPy and other data related libraries
- Experience with Big Data technologies like Spark, Map Reduce, Hadoop, Hive etc
- Proficiencient in data pipeline and workflow management tools e.g. Airflow
- Experience with data visualization tools like Tableau, or PowerBI, AWS QuickSight, etc.
- Experience with the AWS/Azure/GCP data engineering services
- Knowledge of AWS services viz., S3, Lambda, EMR, GLUE ETL, Athena, RDS, Redshift, EC2, IAM OR Knowledge of Azure services viz., ADF, Azure Synapse Analytics, ADLS Gen2, Azure SQL DB
- Very good exposure of working on Data Lakes & Data Warehouses solutions
- Excellent problem-solving, communication, and organizational skills.
- Proven ability to work independently and with a team.
- Ability to guide other data engineers.
- Experience working with data ingestion tools such as Fivetran, stitch, or Matillion.
- Good understanding of NoSQL databases like Redis, Cassandra, MongoDB, or Neo4j.
- Familiarity with machine learning technologies.
- Experience in Java or Scala
Additional Experience Desired:
Benefits
- A competitive annual salary based on experience and market demands
- Flexi-timings
- Work From Anywhere
- Medical insurance with the option to purchase a premium plan or HSA option for your entire family
- Regular Health check-up camps arranged by the company
- Recreational activities (Pool, TT, Wii, PS2)
- Business casual atmosphere
- #LI-Remote
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Agile Airflow Athena AWS Azure Big Data BigQuery Cassandra Data Analytics Data pipelines Data visualization Data warehouse DynamoDB EC2 ELT Engineering ETL Excel FiveTran GCP Hadoop Java Lambda Machine Learning Map Reduce Matillion MongoDB Neo4j NoSQL NumPy Pandas Pipelines Power BI Privacy Python QuickSight RDBMS Redshift Scala Security Snowflake Spark SQL Tableau
Perks/benefits: Career development Competitive pay Health care Team events
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Data Science Intern jobs
- Open Lead Data Analyst jobs
- Open Data Engineer II jobs
- Open Power BI Developer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Science Manager jobs
- Open Marketing Data Analyst jobs
- Open MLOps Engineer jobs
- Open Junior Data Scientist jobs
- Open Data Scientist II jobs
- Open Business Intelligence Developer jobs
- Open Business Data Analyst jobs
- Open Product Data Analyst jobs
- Open Data Analytics Engineer jobs
- Open Data Analyst Intern jobs
- Open Sr Data Engineer jobs
- Open Principal Data Scientist jobs
- Open Sr. Data Scientist jobs
- Open Senior Data Architect jobs
- Open Data Engineering Manager jobs
- Open Junior Data Engineer jobs
- Open Big Data Engineer jobs
- Open Data Quality Analyst jobs
- Open Research Scientist jobs
- Open Azure Data Engineer jobs
- Open GCP-related jobs
- Open Java-related jobs
- Open Data quality-related jobs
- Open ML models-related jobs
- Open Business Intelligence-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open PhD-related jobs
- Open Data visualization-related jobs
- Open Deep Learning-related jobs
- Open Finance-related jobs
- Open NLP-related jobs
- Open PyTorch-related jobs
- Open TensorFlow-related jobs
- Open LLMs-related jobs
- Open APIs-related jobs
- Open Generative AI-related jobs
- Open CI/CD-related jobs
- Open Snowflake-related jobs
- Open Consulting-related jobs
- Open Hadoop-related jobs
- Open Kubernetes-related jobs
- Open Data governance-related jobs
- Open Databricks-related jobs
- Open Airflow-related jobs