Senior/Lead Data Engineer
Gurgaon, Bengaluru, Delhi NCR
Srijan Technologies
Srijan is a digital experience services company that helps global Fortune 500s to nonprofits build transformative digital paths to a better future.Location: Gurgaon, Bangalore, Delhi NCR,None,None
About US:-
We turn customer challenges into growth opportunities.
Material is a global strategy partner to the world’s most recognizable brands and innovative companies. Our people around the globe thrive by helping organizations design and deliver rewarding customer experiences.
We use deep human insights, design innovation and data to create experiences powered by modern technology. Our approaches speed engagement and growth for the companies we work with and transform relationships between businesses and the people they serve.
Srijan, a Material company, is a renowned global digital engineering firm with a reputation for solving complex technology problems using their deep technology expertise and leveraging strategic partnerships with top-tier technology partner. Be a part of an Awesome Tribe
Responsibilities:
- Design and implement scalable and robust ETL pipelines to facilitate the extraction, transformation, and loading of large datasets from various sources into a structured data warehouse.
- Utilize Databricks to process big data workloads, leveraging Spark for data transformation and
aggregation, and optimize performance of big data queries.
- Manage and orchestrate containerized applications using Docker and Kubernetes, ensuring seamless deployment, scaling, and management of applications across clusters.
- Develop and maintain data architectures on Azure, including data lakes, data warehouses, and
other relevant data storage solutions, ensuring security, scalability, and compliance with industry standards.
- Integrate and manage Elasticsearch clusters for efficient searching, logging, and analytics of structured and unstructured data.
- Implement and maintain asynchronous programming models to optimize data processing workflows and improve system responsiveness and scalability.
- Collaborate with AI teams to integrate large language models (LLMs) into data processing pipelines, enhancing data analysis and decision-making capabilities.
- Develop and enforce best practices and standards for data management, including data security, backup procedures, and disaster recovery methods.
- Monitor and troubleshoot performance issues across all platforms and applications, proposing and implementing improvements to system architectures.
- Stay up-to-date with industry trends and advancements in data engineering technologies, applying new techniques and tools as applicable to improve existing systems.
**Must-Have Skills:**
- Advanced proficiency in Python programming, with a strong grasp of software development best practices and efficient code production.
- Deep knowledge and hands-on experience with Azure data services, including Azure Delta Lake, Service Bus, Queues and Topics, and Event Hub.
- Expertise in software design patterns, particularly those relevant to data engineering such as producer-consumer, singleton, and others.
- Proven ability to monitor and scale data pipelines for performance and efficiency, identifying bottlenecks and implementing solutions to enhance data flow and processing.
- Extensive experience with big data technologies, including Databricks, Spark, and Hadoop.
- Proficient in managing and deploying applications using Docker and Kubernetes.
- Solid understanding and experience with Elasticsearch and its deployment and maintenance.
- Experience with asynchronous and concurrent programming models.
- Proficient in developing and managing ETL pipelines for large-scale data.
Why work for Material
In addition to fulfilling, high-impact work, company culture and benefits are integral to
determining if a job is a right fit for you. Here’s a bit about who we are and highlights around
What we offer.
Who We Are & What We Care About:-
• Material is a global company and we work with best-of-class brands worldwide. We also create and launch new brands and products, putting innovation and value creation at the center of our practice. Our clients are in the top of their class, across industry sectors from technology to retail, transportation, finance and healthcare.
• Material employees join a peer group of exceptionally talented colleagues across the company, the country, and even the world. We develop capabilities, craft and leadingedge market offerings across seven global practices including strategy and insights, design, data & analytics, technology and tracking. Our engagement management team makes it all hum for clients.
• We prize inclusion and interconnectedness. We amplify our impact through the people, perspectives, and expertise we engage in our work. Our commitment to deep human understanding combined with a science & systems approach uniquely equips us to bring a rich frame of reference to our work.
• A community focused on learning and making an impact. Material is an outcomes focused company. We create experiences that matter, create new value and make a difference in people's lives.]
What We Offer:-
• Professional Development and Mentorship.
• Hybrid work mode with remote friendly workplace. (6 times in a row Great Place To
Work (Certified).
• Health and Family Insurance.
• 40+ Leaves per year along with maternity & paternity leaves.
• Wellness, meditation and Counselling sessions.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Architecture Azure Big Data Data analysis Databricks Data management Data pipelines Data warehouse Docker Elasticsearch Engineering ETL Finance Hadoop Kubernetes LLMs Pipelines Python Security Spark Unstructured data
Perks/benefits: Career development Health care Parental leave Startup environment
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Data Science Manager jobs
- Open Lead Data Analyst jobs
- Open Marketing Data Analyst jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Engineer II jobs
- Open Principal Data Engineer jobs
- Open MLOps Engineer jobs
- Open Power BI Developer jobs
- Open Data Scientist II jobs
- Open Business Intelligence Developer jobs
- Open Data Analytics Engineer jobs
- Open Business Data Analyst jobs
- Open Sr Data Engineer jobs
- Open Junior Data Scientist jobs
- Open Data Analyst Intern jobs
- Open Product Data Analyst jobs
- Open Sr. Data Scientist jobs
- Open Senior Data Architect jobs
- Open Research Scientist jobs
- Open Big Data Engineer jobs
- Open Azure Data Engineer jobs
- Open Manager, Data Engineering jobs
- Open Principal Data Scientist jobs
- Open Data Quality Analyst jobs
- Open Data Product Manager jobs
- Open Data quality-related jobs
- Open GCP-related jobs
- Open Java-related jobs
- Open Business Intelligence-related jobs
- Open ML models-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open PhD-related jobs
- Open Deep Learning-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open PyTorch-related jobs
- Open TensorFlow-related jobs
- Open APIs-related jobs
- Open NLP-related jobs
- Open Consulting-related jobs
- Open LLMs-related jobs
- Open CI/CD-related jobs
- Open Snowflake-related jobs
- Open Generative AI-related jobs
- Open Kubernetes-related jobs
- Open Hadoop-related jobs
- Open Data governance-related jobs
- Open Airflow-related jobs
- Open Databricks-related jobs