Data Engineer

Data Engineer (Azure Databricks)

Data is at the core of our business.  The Data Engineer serves as a key resource in building and executing on Chubb's information strategy.  The ideal candidate for this role is a self-starter who will take ownership of your projects and deliver high-quality data-driven solutions.  The primary responsibilities include using services and tools to ingest, egress, and transform data from multiple sources.  This includes design, build, and test data pipelines leveraging Azure Databricks. They manage and secure the flow of structured and unstructured data from multiple sources and should be comfortable working with data platforms that can include relational databases, nonrelation databases, data streams, and file stores. Data Engineers also ensure that data services securely and seamlessly integrate with other data platform technologies or application services such as Microsoft Azure or on-premise solutions. 

Data Engineers collaborate with business stakeholders to identify and meet data requirements. They design and implement solutions. They also manage, monitor, and ensure the security and privacy of data to satisfy business needs.

 

Responsibilities

• Analyze business procedures and requirements to recommend specific types of data that can be used to improve upon them.
• Establish work estimates based on business requirements through participation in project feasibility, scoping, and estimating activities. 
• Designing and implementing data storage solutions: This involves designing and implementing storage solutions on Azure, such as Azure Data Lake Storage, Azure Blob Storage, and Azure SQL Database.
• Building and maintaining data pipelines: Responsible for building and maintaining data pipelines to move data from various sources to data storage solutions. This includes building ETL (Extract, Transform, Load) pipelines using Azure Data Factory or Azure Databricks.
• Managing data workflows: Responsible for managing data workflows and ensuring that data is processed in a timely and accurate manner.
• Monitoring data performance: Responsible for monitoring data performance and ensuring that data pipelines and workflows are running smoothly.
• Ensuring data quality: Responsible for ensuring that data is accurate, consistent, and reliable.
• Troubleshooting data issues: Responsible for troubleshooting data issues and resolving them in a timely manner.
• Collaborating with other teams: Responsible for collaborating with other teams, such as data analysts and data scientists, to ensure that their data needs are met.
• Ensuring data security: Responsible for ensuring that data is stored and processed securely, following best practices for data security.
• Ability to gain a complete understanding of systems, data flows, integration points, and quickly assess the impacts of changes.

 

Essential Qualifications

• A bachelor's degree in computer science, information technology, or a related field.
• Strong technical background and proficiency in programming languages like Python, Java, and SQL.
• 3-5 years of relevant work experience, which may include data warehousing, data modeling, ETL development, or database administration. 
• Ability to understand and analyze complex data sets, identify patterns and trends, and develop data models that can support business objectives.
• Prior experience working in the Property and Casualty Insurance domain a must have. 
• Prior experience working in a data warehouse environment or on data integration projects a strong plus.
• Self-motivated and innovative with the ability to set direction, manage own work, and effectively obtain results from others.
• Strong influencing & partnership skills; ability to work collaboratively with business and IT counterparts.
• Exceptional communication & interpersonal skills, verbal & written
• Ability to multi-task & adapt within a constantly changing environment
• Strong organizational & analytical skills, high attention to detail.

Technical Skills

• Strong understanding of Spark architecture and concepts, as well as hands-on experience working with Spark RDDs, DataFrames, and SQL.
• Proficient in either Python, PySpark or Scala, which are the primary languages used to write Spark applications in Azure Databricks.
• Prior experience of working with cloud computing platforms like Azure, AWS, or Google Cloud, and be familiar with the tools and services offered by these platforms, such as Azure Blob Storage, Azure Data Factory, and AWS S3.
• Proficiency with SQL to facilitate the data integration process through data profiling, understanding relationships in the data, and validating the integrity of the data. 
• Experience working with enterprise class RDBMS platforms like Oracle, SQL Server, and DB2. Must have worked on ETL efforts in one of these environments.
• Experience creating UNIX shell scripts to perform file level validation and analysis. 
• Familiarity with NoSQL databases and JSON formats, preferably CosmosDB

Other Knowledge/Skills/Abilities:

• Excellent communication/interpersonal skills, with experience coordinating ETL work with an onshore/offshore vendor team. Must be able to clearly communicate.
• Excellent problem-solving skills. Applies technical knowledge to determine solutions and solve complex problems.
• Proven ability to create and maintain documentation (ETL designs, technical specs, test cases, test results, project status, deployment plans, etc.)
• Experience working on projects using Agile, Scrum, or Scaled Agile Framework (SAFe).