Senior Data Engineer
Remote - Canada - EST
Pythian
Pythian is a global data and analytics services company that helps organizations transform by leveraging data, analytics, and the cloud. We design, implement, and support customized solutions for the toughest data challenges.Why you?Are you a Data Engineer or Big Data Developer who wants to join Pythian’s internal product and tooling team to work on projects including planning, designing and implementing new solutions using the latest Cloud and Big Data technologies? Can you want to work remotely to implement and help develop cutting edge solutions, create data pipelines which will migrate data from on-prem systems and load it into a cloud hosted Enterprise Data Platform? If you Love Your Data™ and solving complex problems then this could be the job for you!
What will you be doing?
- Working with our internal product and tooling team to design and develop end to end Cloud based solutions with heavy focus on application and data, all with good understanding of underlying cloud infrastructure
- Working on complex and varied Cloud Data focused projects such as migrating business-critical applications to the cloud, re-platforming or re-architecting difficult data and analytics use cases; Migrate existing data warehouses from on-premise data center or from one cloud provider to another etc
- Delivering highly reliable software and data pipelines using Software Engineering best practices like automation, version control, continuous integration/continuous delivery, testing, security, etc.
- Define, implement and enforce automated data security and data governance best practices within the solutions designed
- Help our customers to grow from a data warehouse into a true cloud-native data platform with full multi-source data ingestion and integration.
- Build or move analytic workloads to the cloud for better scalability and efficiency.
- Translating complex functional and technical requirements into detailed designs.
- Writing high-performance, reliable and maintainable code.
- Automate, create reusable accelerators to help get the job done quicker and better
- Perform data processing requirements analysis and data flow integrations with external systems.
- Diagnostics and troubleshooting of operational issues. Perform Health-checks and configuration reviews.
What do we need from you?
- While we realise you might not have everything on the list to be the successful candidate for the Data Engineer job you will likely have specialized knowledge and experience in:
- Proficiency in a programming language such as Python, Java, Go or Scala
- Experience with big data cloud technologies like EMR, Athena, Glue, Big Query, Dataproc, Dataflow.
- Ideally you will have specific strong hands on experience working with Google Cloud Platform data technologies - Google BigQuery, Google DataFlow, and Executing PySpark and SparkSQL code at Dataproc
- Understand the fundamentals of Spark (PySpark or SparkSQL) including using the Dataframe Application Programming Interface as well as analysing and performance tuning Spark queries
- Have experience developing and supporting robust, automated and reliable data pipelines
- Develop frameworks and solutions that enable us to acquire, process, monitor and extract value from large dataset
- Have strong SQL skills
- Bring a good knowledge of popular database and data warehouse technologies & concepts from Google, Amazon or Microsoft (Cloud & Conventional RDBMS), such as BigQuery, Redshift, Microsoft Azure SQL Data Warehouse, Snowflake etc.
- Have strong knowledge of a Data Orchestration solutions like Airflow, Oozie, Luigi or Talend
- Have knowledge of how to design distributed systems and the trade-offs involved
- Experience with working with software engineering best practices for development, including source control systems, automated deployment pipelines like Jenkins and devops tools like Terraform
- Experience in data modeling, data design and persistence (e.g. warehousing, data marts, data lakes).
What do you get in return?
- Love your career: Competitive total rewards package with an annual bonus
- Love your development: Hone your skills or learn new ones with our substantial training allowance; participate in professional development days, attend conferences, become certified, whatever you like!
- Love your work/life balance: Why commute? Work remotely from your home (forever), there’s no daily travel requirement to an office! You can be located anywhere in India, all you need is a stable internet connection.
- Love your workspace: We give you all the equipment you need to work from home including a laptop with your choice of OS, and an annual budget to personalise your work environment!
- Love your community: Blog during work hours; take a day off and volunteer for your favorite charity.
DisclaimerFor this job an equivalent combination of education and experience, which results in demonstrated ability to apply skills will also be considered.Pythian is an equal opportunity employer and welcomes applications from people with disabilities. Accommodations are available upon request for candidates taking part in all aspects of the selection process.The successful applicant will need to fulfill the requirements necessary to obtain a background check.Applicants must be legally authorized to work in their country of residence permanently– Pythian will not relocate, sponsor, or file petitions of any kind on behalf of a foreign worker to gain a work visa, become a permanent resident based on a permanent job offer, or to otherwise obtain authorization to work.
Tags: Airflow Athena AWS Azure Big Data BigQuery Dataflow Data pipelines Dataproc DevOps Distributed Systems E-commerce Engineering ETL GCP Google Cloud Machine Learning Oozie Pipelines PySpark Python RDBMS Redshift Scala Security Snowflake Spark SQL Talend Terraform Testing
Perks/benefits: Career development Conferences Gear Salary bonus Travel
More jobs like this
Explore more AI, ML, Data Science career opportunities
Find even more open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general - ordered by popularity of job title or skills, toolset and products used - below.
- Open Business Intelligence Engineer jobs
- Open Lead Data Analyst jobs
- Open MLOps Engineer jobs
- Open Senior Business Intelligence Analyst jobs
- Open Data Engineer II jobs
- Open Sr Data Engineer jobs
- Open Data Manager jobs
- Open Data Analytics Engineer jobs
- Open Principal Data Engineer jobs
- Open Power BI Developer jobs
- Open Junior Data Scientist jobs
- Open Business Intelligence Developer jobs
- Open Product Data Analyst jobs
- Open Senior Data Architect jobs
- Open Data Scientist II jobs
- Open Sr. Data Scientist jobs
- Open Manager, Data Engineering jobs
- Open Business Data Analyst jobs
- Open Big Data Engineer jobs
- Open Data Quality Analyst jobs
- Open Data Analyst Intern jobs
- Open Principal Data Scientist jobs
- Open Data Product Manager jobs
- Open ETL Developer jobs
- Open Junior Data Engineer jobs
- Open Data quality-related jobs
- Open Business Intelligence-related jobs
- Open GCP-related jobs
- Open ML models-related jobs
- Open Data management-related jobs
- Open Privacy-related jobs
- Open Java-related jobs
- Open Finance-related jobs
- Open Data visualization-related jobs
- Open APIs-related jobs
- Open Deep Learning-related jobs
- Open PyTorch-related jobs
- Open Consulting-related jobs
- Open Snowflake-related jobs
- Open TensorFlow-related jobs
- Open PhD-related jobs
- Open CI/CD-related jobs
- Open NLP-related jobs
- Open Data governance-related jobs
- Open Kubernetes-related jobs
- Open Airflow-related jobs
- Open Hadoop-related jobs
- Open Databricks-related jobs
- Open LLMs-related jobs
- Open Data warehouse-related jobs