Technical Architect - Big Data
India Remote
Srijan Technologies
Srijan is a digital experience services company that helps global Fortune 500s to nonprofits build transformative digital paths to a better future.Location: India Remote,,
Job Description:-
Big Data Architect - Kafka
Srijan is hiring for a strong dedicated Professional for the Big Data Architect role. An ideal candidate must have 7+ years of experience in Big Data Technologies and must have good experience of Kafka. Good communication skills, exposure to handling International clients, etc. Please find below the detailed job description and kindly go through the same for reference:-
Role:
A Big Data Architect with Kafka (primary focus) and Hadoop skill sets to work on an exciting Streaming / Data Engineering team (7+ years of total experience)
Responsibilities include:
- Responsible for technical design and implementation in the areas of: big data engineering mainly Kafka
- Develop scalable and reliable data solutions to move data across systems from multiple sources in real time as well as batch modes (Kafka)
- Build Producer and Consumer applications on Kafka, and appropriate Kafka configurations
- Designing, writing, and operationalizing new Kafka Connectors using the framework
- Accelerate adoption of the Kafka ecosystem by creating a framework for leveraging technologies such as Kafka Connect, KStreams/KSQL, Schema Registry, and other streaming-oriented technology
- Implement Stream processing using Kafka Streams / KSQL / Spark Jobs along with Kafka
- Develop both deployment architecture and scripts for automated system deployment in an On-Premise as well as Cloud (AWS)
- Bring forward ideas to experiment and work in teams to transform ideas to reality
- Architect data structures that meet the reporting timelines
- Work directly with engineering teams for design and build their development requirements
- Maintain high standards of software quality by establishing good practices and habits within the development team while delivering solutions on time and on budget.
- Facilitate the agile development process through daily scrum, sprint planning, sprint demo, and retrospective meetings.
- Participate in peer-reviews of solution designs and related code
- Analyze and resolve technical and application problems
- Proven communication skills, both written and oral
- Demonstrated ability to quickly learn new tools and paradigms to deploy cutting edge solutions
- Create large scale deployments using newly conceptualized methodologies
Skills:
- Proven hands-on experience with Kafka is a must.
- Proven hands-on experience with Hadoop stack (HDFS, Map Reduce, Spark).
- Core development experience in one or more of these languages: Java, Python / PySpark, Scala etc.
- Good experience in in developing Producers and Consumers for Kafka as well as custom Connectors for Kafka
- 3 plus years of developing applications using Kafka (Architecture), Kafka Producer and Consumer APIs, Real-time Data pipelines/Streaming
- 2 plus years of experience performing Configuration and fine-tuning of Kafka for optimal production performance
- Experience in using Kafka APIs to build producer and consumer applications, along with expertise in implementing KStreams components. Have developed KStreams pipelines, as well as deployed KStreams clusters
- Strong knowledge of the Kafka Connect framework, with experience using several connector types: HTTP REST proxy, JMS, File, SFTP, JDBC, Splunk, Salesforce, and how to support wire-format translations. Knowledge of connectors available from Confluent and the community
- Experience with developing KSQL queries and best practices of using KSQL vs KStreams will be an added advantage
- Deep understanding of different messaging paradigms (pub/sub, queuing), as well as delivery models, quality-of-service, and fault-tolerance architectures
- Expertise with Hadoop ecosystem, primarily Spark, Kafka, Nifi etc.
- Experience with integration of data from multiple data sources
- Experience with stream-processing systems: Storm, Spark-Streaming, etc. will be ad advantage
- Experience with relational SQL and NoSQL databases, one or more of DBs like Postgres, Cassandra, HBase, Cassandra, MongoDB etc.
- Experience with AWS cloud services like S3, EC2, EMR, RDS, Redshift will be an added advantage
- Excellent in Data structures & algorithms and good in analytical skills
- Strong communication skills
- Ability to work with and collaborate across the team
- A good "can do" attitude.
Tags: Agile APIs AWS Big Data Blockchain Cassandra Data pipelines EC2 Engineering Hadoop HBase HDFS Kafka Machine Learning Map Reduce MongoDB .NET Node.js NoSQL PHP Pipelines PostgreSQL PySpark Python Redshift Research Scala Scrum Spark Splunk SQL Streaming
Perks/benefits: Career development Startup environment
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Business Intelligence Engineer jobs
- Open Lead Data Analyst jobs
- Open MLOps Engineer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Engineer II jobs
- Open Sr Data Engineer jobs
- Open Data Manager jobs
- Open Data Analytics Engineer jobs
- Open Principal Data Engineer jobs
- Open Power BI Developer jobs
- Open Junior Data Scientist jobs
- Open Business Intelligence Developer jobs
- Open Product Data Analyst jobs
- Open Senior Data Architect jobs
- Open Data Scientist II jobs
- Open Sr. Data Scientist jobs
- Open Manager, Data Engineering jobs
- Open Business Data Analyst jobs
- Open Big Data Engineer jobs
- Open Data Quality Analyst jobs
- Open Data Analyst Intern jobs
- Open Principal Data Scientist jobs
- Open Data Product Manager jobs
- Open ETL Developer jobs
- Open Junior Data Engineer jobs
- Open Data quality-related jobs
- Open Business Intelligence-related jobs
- Open GCP-related jobs
- Open ML models-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open Java-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open APIs-related jobs
- Open Deep Learning-related jobs
- Open PyTorch-related jobs
- Open Consulting-related jobs
- Open Snowflake-related jobs
- Open TensorFlow-related jobs
- Open PhD-related jobs
- Open CI/CD-related jobs
- Open NLP-related jobs
- Open Data governance-related jobs
- Open Kubernetes-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open Databricks-related jobs
- Open LLMs-related jobs
- Open Data warehouse-related jobs