Senior Data Engineer

HUMAN was founded in 2012 in a Brooklyn sci-fi bookstore by Tamer Hassan, Michael Tiffany, Dan Kaminsky, and Ash Kalb. Our humble beginnings led to the creation of the Human Verification Engine, the backbone of all our products that protect enterprises from sophisticated bots. Today we verify the humanity of more than 15 trillion interactions per week for some of the largest companies and internet platforms across the internet. 
Our hacker roots still permeate everything we do. You will be a part of the HUMAN front line in our commitment to helping protect companies (and in turn, their customers) from both revenue and reputation risk caused by malicious bots. However, HUMAN is not the center of this story. Humans like yourself are. We firmly believe in putting people first. This approach spans our extensive benefits and day-to-day culture in order for every human to do the best work of their life. We want to hear about the marathon you’re training for. We want to see pictures of your pets. We want to know your favorite robot (we have many). 
The HUMAN R&D team is pivotal to our mission to protect the heart of the internet by disrupting the economics of cybercrime. They are the ones uncovering and fighting bot operations like PARETO, ICEBUCKET, 3ve, and Methbot. HUMAN supports our R&D humans by strongly valuing deep work and flexibility. Company-wide meeting-free Fridays give you uninterrupted time to work on your projects. Additionally, this team is encouraged to participate in R&D Research Friday - a designated half-day each week dedicated to trying new ideas and personal career development. HUMAN is fully committed to flexible working arrangements, or as we call it, work from anywhere, anytime. All of this is to ensure our humans are able to do the best work of their lives. 
We want to work with people like you who break down problems to build up better solutions. That’s what makes us HUMAN.  
The Data Engineering  team is responsible for the primary organs and arteries of our system. The team’s mission is to consistently deliver a world-class data platform that enables our security researchers, threat intel analysts, and data scientists to quickly and reliably detect malicious non-human activity. You will be solving a wide range of problems from finding more efficient ways to research sophisticated botnets to delivering only the most critical and impactful insights to our customers as fast as possible so they can take action.  By joining the Data Platform team at Human you will gain a deep understanding of how our products and services work all the way from signal collection to customer reports and are able to make a major impact on cleaning up the internet.   You will be responsible for the evolution of a platform that will grow in scale by 100x (current volume is over 1 PB) to 1000x, ensure that our data can never be used for ill by anyone (e.g. privacy and security are at the core of the system design), and incorporate the latest ML and fraud research techniques.  

What You'll Do:

• Architect, Design ,Build, Test, Deploy and Maintain reliable, scalable systems and applications to process our current and expected future loads
• Partner closely with security researchers, threat intel analysts, and data scientists to understand how they work and uncover new ways to enhance and amplify their analysis
• Manage the ingestion and usability of a Snowflake data warehouse that ingests and process very high volume near real time data 
• Partner with our operations teams to debug and resolve critical issues and participate in an on-call support rotation for mission critical production applications
• Empower  our stakeholders  to automate and enhance their day to day data requirements (finance, marketing, sales, product management, etc.) leveraging frameworks and libraries.
• Work with a team of uniquely talented engineers who love to learn, experiment, and push the limits of modern technology
• Leverage DevOps Practices and Principles to build and  leverage CI/CD pipelines.
• Participate in Git-based code reviews and design reviews at project to architecture scale
• Participate in building a unique culture that emphasizes lean principles and optimizes for deep work, collaboration, and doing the best work of your career

Who you are:

• You can develop software in  Python , understanding both the theoretical concepts of the language as well as how to write well-documented well-tested code
• You have experience in designing and architecting big data eco systems leveraging large data sets (Order of 100 billion interactions and multiple petabytes) and hands on experience developing them 
• You  have demonstrated ability to design processes that support data transformation, data structures, metadata, dependency and workflow management.
• You have experience efficiently analyzing and extracting key insights from large (terabytes/petabytes) datasets. 
• You have ability to benchmark systems, analyze system bottlenecks and propose solutions to eliminate them
• You have the ability to clearly articulate pros and cons of various technologies and platforms  and to document use cases, solutions and recommendations.
• You can own medium to-large sized features all the way through from technical design to deployment in production
• You have experience actively contributing to a project on a team of 3 or more people and ability to work with cross functional teams
• You can provide updates on the scope and state of work, proactively identify issues that you are facing, and learn to anticipate roadblocks that may arise
• You find time to improve some aspect of a codebase or process through writing small utilities or integrating an open source tool
• These are some additional skills that the team has developed and use regularly but are not required for new hires  (other than a burning desire to learn them when needed):
• Experience with Java
• Experience automating the deployment of distributed software to cloud environments like AWS, Azure or GCP (which likely also means you know some combination of Kubernetes, Docker, and/or Terraform)
• Knowledge of the Snowflake database, primarily its use as a data warehouse 
• Advanced SQL knowledge and working familiarity with a variety of SQL and No-SQL Databases.
• Able to do basic *nix system administration

Life at HUMAN:
HUMAN prides itself on being an equal opportunity workplace. We firmly believe in putting people first regardless of who you are, where you come from, how you identify, or who your favorite robot is (we have many). We are on a mission to protect the integrity of the internet for everyone, so we welcome all individuals to come to share their unique experiences and perspectives as we fight against cybercrime together!
With Humans located in all parts of the world, we’ve fully embraced our diversity of thought and are always looking for innovative ways to connect with one another - even in virtual reality! Although New York City is our HQ, with teams in London, Virginia, and Victoria, we trust our Humans in choosing where they work and how they work. The benefits package we provide reflects our remote-first culture and our commitment to our Humans’ personal career development, which includes annual stipends for home office setup, wellbeing, and learning & development. We also offer weekly lunches, flexible time off, no-meeting Fridays, HUMAN days, sabbatical programs, and so much more.
We’re constantly trying to anticipate the needs of our Humans to ensure each one of us is equally prepared to do some of the best work of our life. Taking care of one another is part of the HUMAN experience and how we build true HUMAN connections.If you are an individual with a disability or special need that requires accommodation, please contact us directly.