24 A/B Testing Interview Questions in Data Science Interviews and How to Crack Them

Here’s everything you need to know about A/B testing interview questions in data science interviews.

24 A/B Testing Interview Questions in Data Science Interviews and How to Crack Them


One of the most common types of data science interview questions from top companies revolves around the humble A/B test. They’re an important test for businesses - A/B tests help businesses decide anything from which product to build to how to design a UI.

The A/B test, also called a split test, is a controlled experiment. You basically split a group of people, like your users or end customers, in half. Offer one half Option A. Offer the other half Option B. Now see which option performs better.

For example, Instagram might want to know if making the Shopping page more prominent would affect the overall spending of users. A team of data scientists would put together a test by selecting a subset of existing Instagram users and testing the original design and the new design to see if the new design offers an improvement in spending.

(Another more recent example is when Instagram changed their feed to be fullscreen - users in the test group hated it!)

A/B tests are critical for companies. This makes it an absolutely vital concept to understand if you’re about to go into a data science interview. You’ll be expected to be confident in the statistical concepts behind an A/B test, how to design a good experiment, and how to interpret the results.

Here’s everything you need to know about A/B testing interview questions in data science interviews.


How to Solve an A/B Test Question


24 A/B Testing Interview Questions in Data Science Interviews and How to Crack Them


I gave a brief description above, but I want to use this section to investigate the concept deeper. When you get asked about A/B tests in your data science interview, your interviewer will want you to demonstrate both conceptual understanding and your business acumen.

Let’s talk about concepts first. A/B testing is A type of statistical and two-sample hypothesis testing. When you do statistical hypothesis testing, you're comparing a sample data set against the population data. Two-sample hypothesis testing is when you use statistics to see if the difference between the two samples is significant or not.

When you set up an A/B test, you need two hypotheses: your null hypothesis, and your alternative hypothesis. Your null hypothesis says any change you observe is purely from chance. Basically, it means you think there will be no difference between your control group and your test group.

Your alternative hypothesis says there will be a statistically significant difference between your control and test group.

To develop your hypothesis, you should use the PICOT method: Population, Intervention, Control, Outcome, and Time.

Here’s a template you can use:

In _______(P), what is the effect of _______(I) on ______(O) compared with _______(C) within ________ (T)?

Here’s that Outlook example in the PICOT format:

In Instagram users (P), what is the effect of making the Shopping page more prominent (I) on spending (O) compared with the unchanged Shopping page (C), over the course of three months (T)?

You will have to develop criteria for your control and test group, too. You probably won’t have the resources to do your A/B test on the entire population of users, so you need to come up with a good way to select a sample population. The key is to sample randomly to ensure you have significant results.

If you accidentally select a group of new users who all happen to be millionaires, that will skew your results independently of the test you’re trying to conduct.

You can use Laerd Dissertation’s great guide to selecting a random sample. Make sure your sample is big enough to be statistically relevant. I recommend Andrew Fisher’s guide on the subject.

Once you create your hypotheses, pick your samples, and run your test, it’s time to analyze your results. Before you recommend making a wholesale change, you want to be absolutely sure it’s worth it.

Let’s say you tested your hypothesis on 1,000 users per group. You find that your control group has an average spend of $50, with a standard deviation of $15. Your test group has a retention rate of $53, with a standard deviation of $16.

That may not seem like very much, but with such large sample sizes you can be extremely confident in rejecting your null hypothesis – that gives us a p-value (two-tailed) = 0.000016, which means there’s only a 0.0016% chance that the outcome is through luck or chance.

There are tons of different factors when it comes to choosing the appropriate test. If you have a small sample size (n<30) you’ll choose a t-test rather than a z-test. If you believe your sample mean could be greater, less than, or the same as your control mean, you’ll choose a two-tailed test. I’m a fan of Antoine Soetewey’s flowchart to make a decision. Reviewing this chart will give you a comprehensive overview of the assumptions behind A/B tests and their designs.


Caveats Around A/B Testing


As you answer these types of questions, you’ll likely be tested on your flexibility of thought. A/B tests are great, but the circumstances often aren’t ideal. Employers will want to know how you handle challenges.

If you’re in B2B, you’ll likely face smaller sample sizes. You may not be able to assume independent populations due to the social network effects. You may not have the resources or time to design a perfect test. There might be times when it doesn’t make sense to run an A/B test.

Interviewers will want to hear how you plan to face those challenges as much as your mastery over statistical tests.

You will face questions such as:

  • How will you plan to compensate for a small sample size?
  • How will you minimize interference between control and test groups?
  • How will you make a test robust if you can’t run it for as long as you want?


Examples of A/B Test Questions


24 A/B Testing Interview Questions in Data Science Interviews and How to Crack Them


Let’s solve a real-world A/B test question example from StrataScratch platform:

Robinhood wants to encourage more people to create an account and plans a campaign where they would give out a few free stocks to new users. Describe in detail how you would implement an A/B test to determine the success of this campaign.

When you answer this type of interview question, you’ll want to address the issues in order:

  • Make any clarifying statements or questions.
  • What does “a few stocks” mean in this instance? Would it be possible to test multiple numbers of free stocks?
  • What metric measures success? Retention, money spent, anything else?
  • Which stock to choose? Or offer any stock of X value?
  • Determine:
    • The length of your experiment. 
    • How you’ll select your sample.
    • The size of your sample.
    • For these questions, there’s no right answer. Just your thought process on how you would determine these.
  • Develop your hypothesis:
    • In new Robinhood users (P), what is the effect of free stocks (I) on your success metric (O) compared with not giving new signups free stocks (C) within your selected time frame (T)?
  • Run the analysis.
  • Measure the results:
    • What test would you use and why? 
    • What qualifies as success?
  • Final thoughts:
    • How would you implement the results of the test?
    • How would you communicate the results to stakeholders?
    • What other tests would you trial?


    24 Examples of A/B Testing Interview Questions You May Face


    I’ve collected several real-world interview questions from around the web to offer an example of the kinds of things interviewers will ask you about this concept. Try to answer them yourself before clicking on the link to get the in-depth answer.

    Experimental Design and Setup

    • What is the goal of A/B Testing? – Hackr.io
    • What are ideal conditions for A/B testing? – Amplitude Blog
    • There are a few ideas to increase conversion on an e-commerce website, such as enabling multiple-item checkout (currently users can check out one item at a time), allowing non-registered users to checkout, changing the size and color of the ‘Purchase’ button, etc., how do you select which idea to invest in? - Emma Ding, Towards data science
    • Company X has tested a new feature with the goal to increase the number of posts created per user. They assigned each user randomly to either the control or treatment group. The Test won by 1% in terms of the number of posts. What do you expect to happen after the new feature is launched to all users? Will it be the same as 1%, if not, would it be more or less? (assume there’s no novelty effect) - Emma Ding, Towards data science
    • We are launching a new feature that provides coupons to our riders. The goal is to increase the number of rides by decreasing the price for each ride. Outline a testing strategy to evaluate the effect of the new feature. - Emma Ding, Towards data science
    • If A/B testing is not an available option, how would you answer a question instead? – Amplitude Blog
    • FB launched a Zoom-like feature. It was generally well-accepted and its usage is growing. You work at Instagram. How would you evaluate if Instagram should add that Zoom-like feature? – u/Objective-patient-37 at Reddit

    Resolving experimentation issues

    • We ran an A/B test on a new feature and the test won, so we launched the change to all users. However, after launching the feature for a week, we found that the treatment effect quickly declined. What is happening? – Emma Ding, Towards data science
    • We are running a test with 10 variants, trying different versions of our landing page. One treatment wins and the p-value is less than .05. Would you make the change?  – Emma Ding, Towards data science
    • How long would you run an experiment? – Amplitude Blog
    • In an A/B test, how can you check if assignment to the various buckets was truly random? – StrataScratch
    • How do you deal with small sample size issues? – Amplitude Blog
    • What issues could impact your A/B test results in the development cycles of our product? – Amplitude Blog
    • How do you mitigate the issues? – Amplitude Blog

    Business-focused questions

    • Tell us about a successful A/B test you designed. What were you trying to learn, what did you learn, and how will the experience help you if you work for us? – Amplitude Blog
    • From your experience with using our product, what improvements would you suggest and what experiments would you set up for them? – Amplitude Blog
    • Let’s say we want to compare Feature A and Feature B in an experiment for a user flow. How would you go about designing this test, given what you know about our product? – Amplitude Blog
    • How do you deal with super long-term metrics where you have to wait two months to get your metric, for example when you try to test for how much money people spend during the two months after seeing a feature? – Amplitude Blog

    Data analysis

    • After running a test, you see the desired metric, such as the click-through rate is going up while the number of impressions is decreasing. How would you make a decision? – Emma Ding, Towards data science
    • What would you do if your experiment is inconclusive, and looks more like an AA test? How would you analyze the test results, and what would you look into? – Amplitude Blog
    • Given data from two product campaigns, how could you do an A/B test if we see a 3% increase for one product? – igotanoffer
    • When you know there is a social network effect and the independence assumption doesn’t hold, how does it affect your analysis and decisions? – Amplitude Blog
    • In our A/B test, the results were not statistically significant. What are some potential reasons for this? – Amplitude Blog

    Future investigations

    • What new hires would you suggest for your A/B testing team if you already have team members for roles X, Y, Z? – Amplitude Blog
    • Which roles on your product team should be involved in your tests, and how would you make it easy for them to be involved? – Amplitude Blog


    Further Resources to Prepare for A/B Testing Data Science Interview Questions


    This A/B test interview question walkthrough paired with 24 example questions should leave you fully equipped to breeze through any A/B test interview question at your data science interview. But if you were unfamiliar with any of the terms mentioned above, or you want more content to review, check out these resources:

    Like any data science interview question, remember to approach these with a sense of curiosity. You’re being tested on your command of statistics and math, but also on your thinking process and your ability to look at the big business picture. Good luck and have fun.

    Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.