https://ift.tt/3dNotoo What is Lift Test? Why is it important? Step by step guide of how to set it up and analyze your result with template...
What is Lift Test? Why is it important? Step by step guide of how to set it up and analyze your result with templates provided.
Among so many different marketing platforms and tactics in the market, it can be overwhelming to choose the best strategy combinations that generate efficient marketing ROIs. Oftentimes, because of the fear of losing out on a potential marketing opportunity, more and more dollars were spent on additional marketing tech stacks or campaigns without knowing if the additional gain is worthy. To avoid wasteful spending, Lift Test is a statistical approach to evaluate your options before allocating budget on a project.
Intro to Lift Test
Lift Test helps identify the incremental return and causal effect of a dependent variable by applying one independent variable to the test group and not to the holdout group.
For example, Lift Test can help you answer how much more the conversion rate (dependent variable) from a set of the audience can be improved by serving ads on Facebook (independent variable) versus not serving ads.
The lift test result is then evaluated by the Hypothesis Testing method. If the lift between the Test and Holdout group is statistically significant, the Alternative Hypothesis is accepted to support that serving ads on Facebook will help the incremental conversion rate, and the Null Hypothesis, serving ads on Facebook will NOT help the incremental conversion rate, is rejected.
What’s the difference between an A/B test vs. a Lift Test?
Lift Test is a type of A/B Test. The essence of a Lift Test is about finding the incremental value of the Test group by not providing the treatment to the Holdout group. Therefore, you are not just measuring the absolute result of a single campaign but quantifying how many more conversions you could not otherwise generate without this additional campaign. On the other hand, A/B Test will provide treatments for both two groups but just different ones, and it’s not used to find the incremental return but the absolute relativity. Therefore, it’s typically called the Control group in the A/B test vs. the Holdout group in Lift Test.
What’s Statistical Significance?
Measuring Statistical Significance is to help you be confident that your test result was not random, and the level of confidence you want to have for the test result is related to the alpha level (Significance Level) you set when calculating the sample size, see the formulas in the Sample Size Calculator Template. Based on the P-Value test result yields (see Significance Calculation Template), if 1- P-Value (Observed Significance) is greater than the target Significance Level, you accept the Alternative Hypothesis and reject the Null Hypothesis.
Lift Test Use Case Examples:
- Is a new campaign worth launching considering people can convert from other existing campaigns?
- What’s my incremental ROI by using a new vendor to help increase reach to prospects?
- What’s my incremental CAC (Customers Acquisition Cost) when offering a discount to incentivize customers to place the first order versus not offering any discounts?
Step by Step Guide to Launch a Lift Test:
1. Outline testing objectives and the testing variable (independent variable) such as serving Facebook ads or not. Note that It’s always recommended to test only one variable at one time to control the data noise.
2. Based on the objectives, you will be able to choose the KPIs you want to use to measure the result including the dependent variable such as conversion rate and the allowable cost per acquisition.
3. Determine if the test will be one-tailed or two-tailed. The type of the test will affect how you calculate the required sample size in the next step. If the lift from the Test group is assumed to be only a positive impact, then it’s a one-tailed test. However, if the lift can cause negative impacts such as over-marketing, which can push away customers to not convert, then it will be a two-tailed test.
4. Determine the minimum required sample size based on different lift levels. First, set the α and β to get the desired significance and power level. The sample size will be different depending on the anticipating holdout group performance, the higher the performance in the holdout group, the smaller the sample size needed from the same level of lift. The same applies to lift level, the higher the lift, the smaller the sample size is required.
Sample size calculator template
5. Based on the minimum required sample size, check if the required lift to meet the desired significance makes sense to your test scenario. If yes, estimate how much the test would cost to collect the required sample size. If the required lift is too high, look for a larger sample size and its required lift to see if achievable. The same thing applies to budget control. If a larger sample size requires cost over budget, then you can choose to go for a higher lift with a smaller sample. It is a balancing act.
6. Prepare the test audience and make sure the sample is randomly selected between the holdout and test group.
For example, you can use Python pandas.DataFrame.sample function to create the randomly generated two groups if using a custom audience list. Note that adding the random_state parameter is useful to reproduce the same random selection for future validation.
Code Example:
#import pandas packages
import Pandas as pd
#Create a subset of audience for the test group
test_female_under_30_ios = df[(df.segment_name == 'female_age_under_30') & (df.ios_user_flag == 1)].sample(n=4442, random_state = 1)
7. Prepare the rest testing materials such as campaign creatives, campaign tracking setup, and so on depending on the test type.
8. Measure the test result by quantifying the lift of the dependent variable such as conversion rate between the holdout and test group to determine if the lift level based on the same size reaches the desired statistical significance.
9. Reject or accept the Null Hypothesis based on the statistical significance. If it is to reject the Null Hypothesis (Accept that Serving ads help), then calculate business profitability metrics to evaluate if the treatment is cost-effective. If the incremental cost per acquisition falls within the allowable range, it means the testing variable (e.g., serving Facebook ads) yields enough incremental conversions (e.g., additional conversions than the holdout group), and it is worth implementing it.
Metrics to calculate Statistical Significance Includes:
Lift Analysis Template
Note: n1= Test Sample Size, n2 = Holdout Sample Size
•Test Conversion Rate: p1
•Holdout Conversion Rate: p2
•Test Standard Error (SE1): SQRT(p1×(1-p1)/n1)
•Holdout Standard Error (SE2): SQRT(p2×(1-p2)/n2)
•Z-Score: (p1-p2)/SQRT(POWER(SE1,2)+POWER(SE2,2))
•P-Value (Excel): 1- NORM.S.DIST(Z-Score, TRUE)
•P-Value (Google Sheet): 1- NORM.S.DIST(Z-Score)
•Observed Significance: 1- P-Value
The above formula is written using the function syntax in Excel and Google Sheet, so it can be directly applied to your Excel/Google Sheet for analysis.
Hope you enjoy the article and find it helpful. Feel free to leave a comment if you have any questions. I would love to hear your feedback!
Marketing Incremental Lift Test 101 was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
from Towards Data Science - Medium https://ift.tt/3pHe9Ec
via RiYo Analytics
No comments