Considerations during a split test
To guarantee that tests are successful and meaningful, there are a couple of things to be mindful of.
Make sure that each study cell spends equally. When you spend more, your cost of Facebook advertising (CPA) naturally increases. If a study cell spends significantly more or less than the others, we cannot compare the results as the CPA is higher due to higher spending.
It is also good to avoid making changes during the test – unless, of course, the changes are the one thing you want to test. For the same reason try to avoid using optimization strategies as they do automatic changes. In some cases, changes are acceptable, though they should be avoided generally speaking.
Dos and don'ts during the ad study This list depends on what is the objective of the test. here are some general guidelines depending on whether it's a test related to audiences, creatives or optimization.
- Dos: You can increase bids and budgets, add or remove creatives, as long as the same changes are made across all study cells.
- Dont's: No updates to the audience should be done, as it will be a new audience and hence a new test.
- Dos: You can increase bids and budgets as long as the same changes are made across all study cells.
- Dont's: No updates to the creatives should be done, as this changes the entire test setup.
Optimization or other tests:
- Dos and don'ts really depend on the test in focus: whether we are comparing two optimization goals or current way of managing bids and budgets to smartly optimization features Or something else, they should be evaluated case-by-case. Discuss do's and don'ts separately with your account manager.
Limitations during the split test
Once the ad study has started, you can:
- rename the ad study or ad study cells
- add new campaigns or ad sets into the ad study cells
The following actions should not be possible according to Facebook, but we have found out they can be done:
- remove campaigns from the ad study cells
- remove ad study cells
In theory, you could also add new ad study cells into the ad study, but you can't change the percentages of existing cells, and the sum of percentages cannot exceed 100%. Pay extra attention when you are adding or removing campaigns or ad sets from an active ad study. The audience split will be lifted from those campaigns, and their ads will start showing to 100% of the targeted population. You can also extend the duration of the ad study by updating the end time, if the study has not yet ended. However, while the ad study is already running, you can not
- change the percentages of the existing ad study cells
- add or modify objectives in a lift study
Ending the split test
The ad study ends automatically when the end time is reached. If you want to extend the duration of an ad study, you can edit the end time to some future date as long as the ad study is running.
To end an ad study immediately, click the Show results link for the ad study and use the End ad study button at the bottom.
Notice that this action cannot be undone.
Allowing the test to run to get results
Allow the study to run until there are enough conversions
The Statistical significance calculator reports whether you still need more conversions:
Just because Campaign A received 14 conversions and Campaign B only 10 does not mean that Campaign A is better: the difference might be due to random variation, instead of an underlying true difference in performance. The only way to tell the two apart is to collect enough data. When you create an ad study in Smartly.io you will automatically see an estimate on how much data you are likely to need. When the ad study is running, we will tell you if the ad study can be stopped. You can read more about statistical significance here.
Most people tend to underestimate the magnitude of random variation. A good rule of thumb is that differences smaller than 2 ⋅ √CV are not statistically significant (CV = number of conversions). For example, if one campaign receives 30 conversions and the other 37, the difference is not statistically significant (7 < 2√30 ≈ 11) . You would need at least 41 conversions in the second campaign until the difference would be even close to statistically significant – and that would require the second campaign to be almost 40% better than the first one.
Run the ad study at least one week or the duration of your sales cycle Also, just because you have enough conversions does not mean that the test has been running long enough. You should run the study long enough for ads to have an effect. As a rule of thumb, the ad study duration should be as long as your sales cycle is. Performance can differ between weekdays, especially between working days and the weekend. By running the ad study at least one full week, you get a better idea about its performance.
FAQ — Frequently Asked Questions
Does the ad set reach estimate reflect audience size with or without an ad study?
Without — the reach estimate always shows how big the audience would be without the ad study.
Will the cells have identical delivery?
Unfortunately, not always. Facebook will split the audience according to your split settings, but the ads within the campaigns will go normally through the auction as every other campaign. Therefore it can happen, through randomness, that one split cell gets slightly more delivery than the other.
What happens to the audience when the ad study ends?
When the ad study ends, the audience split no longer applies and both campaigns are shown to the entire audience. Notice that this may lead to campaigns having overlapping audiences. Typically, you want to pause one or both campaigns after the ad study ends.
How can I stop/end my ad study?
You can change the ad study's end date to a very near future date and time (e.g. a minute from now), and it will end then.
Remember to pause unneeded campaigns (typically those contained in the losing cell), so that you don't run them unnecessarily.
Can I reactivate an ad study that has ended?
No. If you need more data to reach conclusive results, we recommend creating a new ad study.