Senior Data Engineer
Pakistan
Applications have closed
10Pearls
10Pearls | The leading IT, Software, Web, App, and Emerging Technologies Services & Solutions | Enabling & Transforming Digitally Fortune 500 Clients WorldwideRequirements:
We are looking for a “Data Engineer”. Ideal candidate should have a Bachelor’s degree in Computer Science with 3 – 5 years of programming experience on Python, C/C++, Java, Perl, Golang, or other such languages. Strong knowledge of database solutions is a must.
Responsibilities
- Develop, construct, test and maintain optimal data pipeline architecture
- Assemble large, complex data sets that meet functional / non-functional business requirements
- Identify ways to improve data reliability, efficiency and quality
- Prepare data for predictive and prescriptive modeling
- Use data to discover tasks that can be automated
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Monitoring processes performance and advising any necessary changes
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Create data tools for analytics that assist in building and optimizing the product into an innovative industry leader
- Work with data and analytics experts to strive for greater functionality in our data systems.
- Demonstrate proficiency in data management and automation on Spark, Hadoop, and HDFS environments
- Understanding database management, and as such, in-depth knowledge of SQL is required. Likewise, other database solutions, such as Mongo, Cassandra or Bigtable, are good to have.
Requirements
Required Skills
The candidate must have,
- Good communication skills
- Experience with Apache Hadoop, Hive, Spark, Airflow, Sqoop, Snowflake, Apache Livy, Delta Lake
- Experience in AWS EMR (HDFS, S3, Hbase), AWS Athena, PySpark or related technologies
- Experience with streaming technologies such as kafka
- Experience of working with programming languages like Scala, Java, SQL, Python, R etc.
- Experience with IAC and automation frameworks such as Terraform, Ansible, Jenkins and Packer
- Proficient in event-based architectures (Kinesis, Kafka, Confluent, etc.), REST interfaces, data pipelines and other real-time strategies
- Ability to dig deeper into the issues of the production critical systems and provide permanent fixes for the system
- ML : Foundational Concepts / MLOps : Foundational Concepts + Experience IAC : CDK, CFN or SAM
- Experience in managing data in relational databases and developing ETL pipelines
- Exposure to enterprise level service such as Cloudera, Databricks, AWS, etc
- Exposure to AWS data services and technologies such as EC2, EMR, Kinesis, Lambda, DynamoDB are nice to have
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Airflow Ansible Athena AWS Big Data Bigtable C++ Cassandra Computer Science Databricks Data management Data pipelines DynamoDB EC2 ETL Golang Hadoop HBase HDFS Kafka Kinesis Lambda Machine Learning MLOps Perl Pipelines PySpark Python R RDBMS Scala Snowflake Spark SQL Streaming Terraform
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Marketing Data Analyst jobs
- Open MLOps Engineer jobs
- Open Junior Data Scientist jobs
- Open AI Engineer jobs
- Open Data Engineer II jobs
- Open Senior Data Architect jobs
- Open Sr Data Engineer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Analytics Engineer jobs
- Open Power BI Developer jobs
- Open Manager, Data Engineering jobs
- Open Product Data Analyst jobs
- Open Principal Data Engineer jobs
- Open Business Data Analyst jobs
- Open Data Quality Analyst jobs
- Open Data Manager jobs
- Open Sr. Data Scientist jobs
- Open Data Scientist II jobs
- Open Big Data Engineer jobs
- Open Business Intelligence Developer jobs
- Open Data Analyst Intern jobs
- Open Principal Data Scientist jobs
- Open ETL Developer jobs
- Open Azure Data Engineer jobs
- Open Data Product Manager jobs
- Open Business Intelligence-related jobs
- Open Data quality-related jobs
- Open Privacy-related jobs
- Open Data management-related jobs
- Open GCP-related jobs
- Open Java-related jobs
- Open ML models-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open Deep Learning-related jobs
- Open APIs-related jobs
- Open PyTorch-related jobs
- Open PhD-related jobs
- Open Consulting-related jobs
- Open TensorFlow-related jobs
- Open Snowflake-related jobs
- Open NLP-related jobs
- Open Data governance-related jobs
- Open Data warehouse-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open Databricks-related jobs
- Open LLMs-related jobs
- Open DevOps-related jobs
- Open Kubernetes-related jobs