Lead Data Engineer, Kafka
Bengaluru
Applications have closed
ASAPP
Elevate human performance using the power of AI. Achieve breakthrough results in customer experience by empowering your agents with integrated automation.What you'll do
- Manage large scale multi-nodes Kafka cluster environments residing on AWS
- Handle all Kafka environment upgrades, including capacity planning, cluster setup, performance tuning and ongoing monitoring
- Perform high-level, day-to-day operational maintenance, support for the Kafka Cluster
- Recognize data flow patterns and generalizations to automate as much as possible to drive productivity gains
- Expand our logging and monitoring processes to discover and resolve anomalies and issues before they become problems
- Develop state-of-the-art automation and data solutions in Python, Spark and Flink
- Maintain, Manage, Monitor our infrastructure related including Kafka, Kubernetes, Spark, Flink, Jenkins, general OLAP and RDBMS databases, S3 objects buckets, permissions
- Know how to make the tradeoffs required to ship without compromising quality
What you'll need
- 8+ years of experience in general software development and/or dev-ops, sre roles.
- 5+ years experience in data engineering, data systems, pipeline and stream processing.
- Expertise in at least one flavor of SQL, e.g. Redshift, Postgres, MySQL, Presto, Spark SQL, Hive
- Proficiency in a high-level programming language. We use Python, Scala, Java, Kotlin, and Go
- Experience with CI/CD (continuous integration and deployment)
- Experience with workflow management systems such as Airflow, Oozie, Luigi, and Azkaban
- Confidence operating in a devops-like capacity working with AWS, Kubernetes, Jenkins, Terraform, etc. thinking about automation, alerting, monitoring, and security and other declarative infrastructure
What we'd like to see
- Bachelor’s Degree in a field of science, technology, engineering, or math, or equivalent hands-on experience
- Experience in maintaining and managing kafka (not just using). Kafka Admin experience is a huge plus
- Experience in maintaining and managing OLAP/HA database systems (not just using)
- Familiarity handling Kubernetes clusters for various jobs, apps, and high throughput
- Technical knowledge of data exchange and serialization formats such as Protobuf, Avro, or Thrift
- Experience in either deploying and creating Spark Scala and/or Flink applications
Benefits
- Competitive compensation
- Stock options
- ICICI Lombard General Insurance LTD
- Onsite lunch & dinner stipend
- Connectivity (mobile phone & internet) stipend
- Wellness perks
- Mac equipment
- Learning & development stipend
- Parental leave, including 6 weeks paternity leave
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Airflow Avro AWS Azkaban CI/CD CX DevOps Engineering Flink Java Kafka Kubernetes Machine Learning Mathematics MySQL NLP OLAP Oozie Pipelines PostgreSQL Python RDBMS Redshift Scala Security Spark SQL Terraform
Perks/benefits: Career development Competitive pay Equity Insurance Parental leave Wellness
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Marketing Data Analyst jobs
- Open MLOps Engineer jobs
- Open AI Engineer jobs
- Open Junior Data Scientist jobs
- Open Data Engineer II jobs
- Open Senior Data Architect jobs
- Open Sr Data Engineer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Analytics Engineer jobs
- Open Power BI Developer jobs
- Open Manager, Data Engineering jobs
- Open Product Data Analyst jobs
- Open Principal Data Engineer jobs
- Open Business Data Analyst jobs
- Open Data Quality Analyst jobs
- Open Data Manager jobs
- Open Sr. Data Scientist jobs
- Open Data Scientist II jobs
- Open Big Data Engineer jobs
- Open Business Intelligence Developer jobs
- Open Data Analyst Intern jobs
- Open Principal Data Scientist jobs
- Open ETL Developer jobs
- Open Azure Data Engineer jobs
- Open Data Product Manager jobs
- Open Business Intelligence-related jobs
- Open Data quality-related jobs
- Open Privacy-related jobs
- Open Data management-related jobs
- Open GCP-related jobs
- Open Java-related jobs
- Open ML models-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open Deep Learning-related jobs
- Open APIs-related jobs
- Open PyTorch-related jobs
- Open PhD-related jobs
- Open TensorFlow-related jobs
- Open Consulting-related jobs
- Open Snowflake-related jobs
- Open NLP-related jobs
- Open Data governance-related jobs
- Open Data warehouse-related jobs
- Open Databricks-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open LLMs-related jobs
- Open DevOps-related jobs
- Open CI/CD-related jobs