Staff Data Scientist, Inference - Experimentation

Controlled Experiments are a key part of product development and data-informed decision-making at Airbnb. Our A/B testing platform, the Experiment Reporting Framework, is used by over 60% of all engineers, data scientists, and product managers every week, in every major product area at the company, with more than 1,000 experiments in flight at any given time.

The Experimentation Data Science team aims to maximize the effectiveness of our experiments by developing better experimentation methodologies, designing and analyzing the most interesting experiments across the company, and partnering with the experimentation platform engineering team to build a world-class experimentation platform. We are looking for a Staff Data Scientist who can uplevel and increase trust in experimentation by analyzing our most challenging experiments, introducing new methods to address potential biases, and making it easier to get trustworthy results. Our team is headed by VP and Technical Fellow Ronny Kohavi, co-author of Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing.

The position is based in our San Francisco office. You can start to work remotely, but we ask (and will help with) relocation to the San Francisco Bay Area by August 2021.

Responsibilities:

• Work closely with engineering and data scientists to design and implement new features and methodologies for experimentation. For example incorporating multiple testing, variance reduction techniques, interaction detection.
• Work with data scientists across the company to help solve challenging problems in experimentation design and implementation.
• Research challenging experimentation questions to inform our approach to experimentation. Example projects include how to extrapolate from short-term proxy metrics to long-term impact and how to measure impact of large-scale changes (e.g. the impact of changing our webpage design system). 
• Mentor and partner with embedded data scientists on individual teams to share best practices and address their key experimentation problems.

Qualifications: 

• 5+ years of industry experience and either a PhD in Statistics (or equivalent) and strong engineering knowledge or MS in Computer Science (or equivalent) with strong statistical knowledge.
• Significant industry experience in controlled experiments.
• Proficient in R or Python, as well as query languages such as SQL, Hive or Presto.
• Experience working on production code using Java/Scala/Ruby, strong understanding of engineering systems and web applications, and full stack experience is a plus.

Recent blog posts about experimentation and papers from our team at Airbnb:

• Selection Bias in Online Experimentation
• Scaling Airbnb’s Experimentation Platform
• Improving Deep Learning for Airbnb Search
• Managing Diversity in Airbnb Search

Benefits:

• Stock
• Competitive salaries
• Quarterly employee travel coupon
• Paid time off
• Medical, dental, & vision insurance
• Life insurance and disability benefits
• Fitness Discounts
• 401K
• Flexible Spending Accounts
• Apple equipment
• Commuter Subsidies
• Community Involvement (4 hours per month to give back to the community)
• Company sponsored tech talks and happy hours
• Much more...