Principal Software Engineer - Big Data
Pakistan
Applications have closed
NorthBay Solutions
Get access to all of our AWS best practices and insights. Subscribe NowYou will have the chance to join a global, multi-cultural team of Data Engineers, and be part of the growth and development of it, participate in a large variety of exciting and challenging projects and work with our exceptional team of Technology professionals.
Technology Stack Used & Required Knowledge:
- Over 6+ years of experience on ETL Data Engineer, Python, Java, Bigdata (Hadoop, spark, hive)
- Excellent understanding of ETL cycle.
- Experience in creating and driving large-scale ETL pipelines in AWS-based environment
- Experience with integration of data from multiple data sources.
- Good knowledge and understanding of Big data engineering, ETL, Data Modeling, and Data Architecture.
- Strong software development and programming skills with a focus on data using Python/ PySpark and/or Scala for data engineering.
- Experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Lambda, Athena, and Redshift will be a plus
- Understanding of data types/ handling of different data models.
- Experience with the AWS data management tools such as Data lake and Databricks or AWS Snowflake is a plus
- Understanding of descriptive and exploratory statistics, predictive modelling, evaluation metrics, decision trees, machine learning algorithms is a plus
- Good scripting and programming skills.
- Should have experience in Jira and Agile development methodologies.
- Must be open to learn and the ability to grasp new technologies quickly.
- Be motivated and passionate about technology.
- Team player with strong integrity and professionalism who can work well with others in the group.
- Self-starter with a proven ability to take initiative and strong problem-solving skills.
To keep it short, below are some key responsibilities:
- Design and develop using Big Data technologies.
- Design and develop cutting edge Big Data, and Cloud technologies to build critical, highly complex distributed systems from scratch
- Design, develop and manage complex ETL jobs and pipelines.
- Act as a team member, consulting other teammates as your new systems transform the way this entire firm leverages data
- Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.
- Support and helping them learn and improve their skills, as well as innovate and iterate on best practices.
- Solve complex issues and provide guidance to the team members when needed.
- Make improvement and process recommendations that have an impact on the business.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Agile Architecture Athena AWS Big Data Consulting Databricks Data management Data warehouse DevOps Distributed Systems EC2 Engineering ETL Hadoop Java Jira Lambda Machine Learning Pipelines PySpark Python Redshift Scala Snowflake Spark Statistics
Perks/benefits: Career development Competitive pay Health care
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open MLOps Engineer jobs
- Open Lead Data Analyst jobs
- Open Data Science Manager jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Engineer II jobs
- Open Data Manager jobs
- Open Sr Data Engineer jobs
- Open Principal Data Engineer jobs
- Open Data Analytics Engineer jobs
- Open Power BI Developer jobs
- Open Business Intelligence Developer jobs
- Open Junior Data Scientist jobs
- Open Data Scientist II jobs
- Open Senior Data Architect jobs
- Open Product Data Analyst jobs
- Open Sr. Data Scientist jobs
- Open Business Data Analyst jobs
- Open Manager, Data Engineering jobs
- Open Big Data Engineer jobs
- Open Data Analyst Intern jobs
- Open Data Quality Analyst jobs
- Open Data Product Manager jobs
- Open Azure Data Engineer jobs
- Open Junior Data Engineer jobs
- Open ETL Developer jobs
- Open Data quality-related jobs
- Open Business Intelligence-related jobs
- Open ML models-related jobs
- Open GCP-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open Java-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open APIs-related jobs
- Open Deep Learning-related jobs
- Open PyTorch-related jobs
- Open Consulting-related jobs
- Open Snowflake-related jobs
- Open TensorFlow-related jobs
- Open PhD-related jobs
- Open CI/CD-related jobs
- Open NLP-related jobs
- Open Kubernetes-related jobs
- Open Data governance-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open Databricks-related jobs
- Open LLMs-related jobs
- Open DevOps-related jobs