If you want to create a Lift Test to measure the impact of your advertising campaigns, please contact your Smartly.io Account Manager to get started.
Conversion Lift Tests, or just lift tests or lift studies, allow you to measure how many of the conversions you get are actually caused by your Facebook ads. While it is next to impossible to find the cause of any individual conversion (why did you buy a new phone — because you saw an ad or because a friend recommended it?), we can measure the total effect quite accurately with a randomized controlled trial. At a high level lift tests work like this:
- The audience is split into treatment group that is targeted normally and control group the does not see any ads.
- Facebook measures the total number of conversions in both groups.
- Incremental conversions in the treatment group are caused by your ads.
Note that Conversion Lift Tests allow you to measure conversions from people who did not even see any ads. Normal reporting is based on last-click attribution and only allows you to see conversions that happened within 28 days after click or view.
The most common obstacle to running Lift Tests is that in order to get useful insights you need to collect enough data, in some cases 2000 conversions or more. This can mean that you need to run the Lift Test for several months, during which nobody in the control group can see your ads. More information in Lift Testing Best Practises.
Please note that Smartly.io currently supports only conversion Lift Tests. Brand Lift Tests – Lift Test with an additional questionnaire to Facebook users to measure brand recognition and recollection – can only be created with Facebook's support.
How to create a Lift Test in Smartly.io
NOTE: Before creating Lift Tests, you must:
1) Select a Business Manager for your ad account in Account Settings.
2) Connect your Facebook account to your Smartly.io user.
You can create Lift Test in Library → Ad Studies. Click Create Ad Study, then select Lift Test. (If you cannot see the selection please contact your Smartly Customer Success Manager.)
Duration defines how long the lift study lasts. The start and end dates, or the duration, can be edited as long as they are in the future: when the date has passed, it can no longer be edited. You can still edit the end date while the test is running, to either extend the duration or end the lift study earlier. However, once the lift study ends it is no longer possible to extend it. Typically you want to set the end time far into the future, and edit it later if needed. The time between the start and end times is called the test period.
The observation period takes place after the test period ends. It does not stop the delivery of your campaigns, but you should do that yourself. During the observation period, no new users are added to the study population. Conversions will be measured from those who were added to the population during the test period (in the control group, users who were considered for delivery during the test period).
The purpose of the observation period is to make sure that late conversions caused by the campaign are also recorded and attributed to the test. Typically the duration of the observation period should not be more than a few days.
Note: Facebook has deprecated what used to be called the Cooldown Period, a quiet period before the test started. To replicate a cooldown period yourself, pause all ads toward the test audience for some time (how long?) before the test starts, and activate them at the test start time. The purpose of this is to wait until any old ads' effect on the control group fades away.
The meaning of Observation period has also changed: ads used to be stopped automatically during the Observation Period; now, you have to stop them manually.
When the Lift Test is running the treatment group is delivered normally but the control group does not see any ads from campaigns included in the Lift Test. Make sure you do not accidentally target the same audience in campaigns that are not included in the Lift Test. Otherwise, the control group will not be a proper control, and the measured effect will appear smaller!
Power analysis can be used to estimate how many conversions you are likely to need in order to get a statistically significant result. The estimate depends on three factors:
- The estimated actual lift your campaign has. We ask you to estimate this beforehand, because it helps estimate the time and budget needed for the test: a small lift effect is way harder to find than a big lift effect.
- The fraction of the treatment group reached by your campaigns during the Lift Test. Reaching more of the treatment group makes it easier to get results, as you have more data points. However, don't try to reach everyone: leave room for Facebook's algorithms to only pick the most optimal people, for optimal results.
- The split between treatment and control group. A 50/50 split makes it easiest to get results, but also means that half of the audience will not see your ads during the Lift Test.
Read about power analysis in more detail: Lift Test power analysis
The simplest Lift Test has only one study cell split into a treatment group and a control group. This setup allows you to test how big of a lift is caused by the test campaign(s) compared to not running them. If you want to measure the lift of all campaigns, you can include the entire ad account in the cell.
In general, you can create one or more cells and assign them treatment and control percentages. Then you choose which campaigns, ad accounts or ad sets for each cell.
If you have more than one cell you will in effect get a combined Split & Lift Test. This setup allows you to both measure the lift in each cell, as well as compare the cells' lift against each other. However, you will have less audience in each cell and therefore it takes more time to collect enough data to get a statistically significant result.
The sum of treatment and control group sizes across all cells must be 100%.
In order to measure lift we need to know not only how many conversions we get from those who see the ads, but also how many conversion we get from those who do not see the ads. Normally Facebook does not record the latter, which is why you need to tell Facebook beforehand what you want to measure.
Each objective records the results for one or more conversion event. Lift Test results are available separately for each objective. Objective type is either Sales, Non-Sales or Mobile App Installs. For Non-Sales objective you get results for the number of buyers and the number of conversions. For Sales objectives you get results also for the amount of sales revenue. For Mobile App Installs objectives, you simply define your app, and Facebook measures the number of app installs.
You can have as many objectives as you want and typically you want to create a separate objective for each conversion event in the sales funnel.
If you include multiple conversion events in a single objective the result will include the sum of all events, and it is not possible to split those afterward. However, the combined objective might give you statistically significant results earlier as it has more data. The typical use case for this is when you have both in-app and web sales, and you are interested in the total amount of sales. In addition to this combined objective you can also create an objective for each conversion event separately to get results also for each individual event.
When specifying an objective make sure you choose an event from the correct pixel or app (see screenshot below). Alternatively, you can combine events from multiple pixels or apps into a single objective, as described above.
Lift Test results are available by clicking the Show details link in Library → Ad Studies.
You can see results when there are at least 100 conversions in the treatment and control groups combined. However, 100 conversions is rarely enough to get statistically significant results! Results are updated daily, so if you cannot see results now check again tomorrow.
Here are few tips to help make sense of the numbers:
- Because the treatment and control group can have different sizes, the number of conversions in the control group is scaled up to make the numbers comparable.
- The number of additional conversions tells you exactly how many conversions you got because of your ads. This is the actual causal effect of your ads.
- Typically you do not reach everyone in the treatment group, which means that there are people in the treatment group who have converted, even though they did not see your ads.
- There might even be people in the "reached treatment group" who first converted, and only then saw your ad. They are still included in the result for statistical reasons.
Lift percentage is calculated relative to the number of people who saw your ads and would have converted without seeing the ads. Thus for example, 100% lift means that half of the people you reached converted because of your ads. Or, that you got 100% more conversions, thanks to your Facebook ads, than you would have gotten if you had not run those ads.
Note that one Lift Test measures the lift of the specific combination of campaigns, audiences and creatives that were included in the Lift Test. Be careful not to generalize the results to other kinds Facebook campaigns. If the lift is smaller than you expected, you might want to rethink the type of creatives you use or the audience you target.
For example, retargeting campaigns usually target high-intent consumers and get great results in normal reporting that is based on last-click attribution. However, lift is usually much lower, indicating that most people would have converted anyway, regardless of your ads.
Lift Test Timeline
In addition to showing you the current results, Smartly.io can also show a timeline of how the results changed over time. The estimates for baseline and incremental conversions are shown daily on a graph, so you can see how the lift results progressed over time. Note that we can only record this history from the time that the Ad Study was synced to Smartly. io – if you import an Ad Study by ID, we'll start recording the history from that moment on. Read more: Lift Test Timeline
Read more in-depth about Lift Testing Best Practises
Smartly.io Blog: How to Drive Incremental Revenue on Facebook (February 2018)