You can create new a new Facebook split test in the Ad Studies page. It can be found in three ways:
- The three-dot Navigation menu next to the account selector in Accounts main page - Campaign Assets - Ad Studies
- The three-dot Navigation menu next to the account selector in Campaigns - Campaign Assets - Ad Studies
- Account Library submenu - Campaign Assets - Ad Studies
Click Create Ad Study and choose Split Test. This opens a pop-up where you will input the details.
First, you must set up the test structure and basic information: name, start time and duration. You can edit the start time if the test has not yet started (useful for creating drafts). Set the duration according to how long it takes to get the required amount of conversions typically. You can edit the duration/end time if the test has not yet ended.
2. Power analysis
You also need to estimate how much data you are likely to need to reach a statistically significant result for your Facebook split test. The estimation tool is built into the ad study creation dialog in Smartly.io, and you need to supply a few details to get a reliable estimate. The default settings are designed to work in most cases, but you can also change them if needed.
- The Metric and Conversion goal define what you want to measure. These are used as default settings when you view the results of the ad study, and can be easily changed at any point. In most cases, the metric and event should be based on your business KPI (eg. CPA for purchase). The selection here is also used as default when estimating the statistical significance of the test, once the results start coming in. The initially selected conversion goal is your primary account reporting goal. You can change it in Settings → Reporting.
- The Smallest interesting difference is the most important factor in calculating how much data you will need to collect. The smaller the differences you want to find, the more data you will need to collect to distinguish the differences from random variation. In most cases, values between 10%–20% give the best compromise between the price of the ad study and the value gained from learnings.
- The Confidence level defines how certain you want to be that the difference you find is a true difference, and not a result of random variation. If there is no difference at all between the ad study cells, and the confidence level is set to 95%, there is a 5% probability of a false positive, or that a statistically significant difference is detected anyway. A larger value means that the outcome is more likely to be correct, but also that more data must be collected to reach a statistically significant result.
- Because randomness is at the core of statistical testing, it is not possible to predict exactly how much data is needed. Sometimes, you get lucky and 300 conversions are enough, but you might need 800 in most cases. The Statistical power allows you to explore this uncertainty in advance. The default value of 80% means that by collecting the indicated number of conversions you will get a statistically significant result with 80% probability (assuming the difference is exactly as large as smallest interesting difference). In this case, you would need to collect more data with a 20% probability. Note that the statistical power is only used to calculate the estimate, and does not affect the actual calculation of statistical significance at the end of the ad study.
- CPA can be filled in to estimate the total cost of the ad study. If you have already run a campaign that is similar to the one you are planning to test, the CPA from the previous campaign might be a good estimate here. Remember to account for late conversions: with long attribution windows it might take a while after the impression event for the conversion to happen
The displayed number of required conversions (or clicks in case of CTR) is always the total number of conversions in all cells. If you add new cells, the estimate will increase accordingly. The estimate is calculated assuming the total budget for the ad study is split proportional to the cell sizes.
3. Study cells
- Add study cells for each variation you want to test. Each cell will have a unique, non-overlapping split of the target audience - You can define the size (% of target audience) of each cell, but the sum needs to be smaller or equal to 100%.
- Give your cells descriptive names. This will make it easier to read the results, as they are shown by ad study cell.
- When the structure is set up, click on the Add items button on each cell to add adsets, campaigns or even full accounts to each cell. Usually, you want to compare only one feature at a time (such as the effect of bidding type) while keeping everything else identical. The easiest way to achieve this is to first create one campaign or adset, then clone it and modify whatever feature you want to test in the new cell.
- You can add one or multiple campaigns, adsets or accounts to each cell. Accounts, campaigns or adsets in the same cell will use the same segment of the split audience, and thus it is possible that their audiences overlap within the cell.
- When you are done, click Save to save the ad study. When your Facebook split test starts, the target audience of each campaign will be limited to the selected segment.
Can I create an ad study with two different DPA campaigns?
Yes. This Facebook split testing setup is a good way to test differences in CTR (or conversions) between two Dynamic Image Templates, for example. Our tests with clients have shown that there can be major differences in performance between different image templates.
Note: If you create an ad study with DPA campaigns that have different dynamic targeting rules, people will not "jump" from one audience to the other. So for example, if one audience includes people who have viewed a product within 1-7 days and the other 8-14 days, a person in the first audience will not jump to the second audience after 8 days have passed, due to the Facebook audience split being still in effect.
How do you define "smallest interesting difference"?
You could also say: "How big do you think the difference will be?" We ask this because it makes a big difference in the estimated test duration. Big differences are easy to find with little data, but small differences require lots of data to detect.
We define the smallest interesting difference as the total difference relative to mean: if CPA is $20 in Cell A and $25 in Cell B, the relative difference is $5 / $22.5 ≈ 22.2%. This definition is used because usually, you do not know a priori which cell is better, so it makes sense to use a symmetric difference.
An alternative would be to define the smallest interesting difference relative to a known control cell. This makes sense when you compare as a new setup against an old setup. For example, if Cell A would be the old setup used as a control, then the relative difference could be defined as $5/$20 = 25%.
There is actually a simple mapping between these. If X is the smallest interesting difference according to the first definition, and Y according to the second, then X = 2Y/(2+Y) and Y = 2X/(2-X). In other words, if you want to use the smallest interesting difference of 20% relative to the control cell (Y = 0.2), you can set the value to 18.2% in the selection.