Incrementality testing: measuring the sales your ads actually caused
Incrementality is the sales that would not have happened without the ad.
An incrementality test measures it by withholding the ad from a group of people and comparing their purchases with the people who saw it.
Why it matters
Every ad platform reports conversions, and every one of them counts people who would have bought anyway. Someone who searched your brand name and clicked the ad at the top was going to find you. Someone who bought after seeing a retargeting ad had already put the item in their basket. The platform counts both as conversions it drove. The incremental figure is smaller, sometimes much smaller.
The decision it changes is whether to keep spending on a channel at all, and at what level. A channel with a platform-reported cost per order of £6.00 and an incremental cost per order of £20.00 is a different proposition, and only the second number tells you which.
How it works
Take the people you would normally show the ad to. Split them at random. Show the ad to most of them (the test group) and withhold it from the rest (the control, or holdout, group). Because the split was random, the two groups are alike in every way except the ad, so any difference in their purchase rate is the ad's doing.
Here is an illustration with made-up numbers.
| Test group | Control group | |
|---|---|---|
| People | 90,000 | 10,000 |
| Orders | 2,700 | 250 |
| Order rate | 3.00% | 2.50% |
The control group bought at 2.50% with no ads. If the test group had bought at that rate too, it would have placed 90,000 × 2.50% = 2,250 orders, against the 2,700 it actually placed. The difference, 450 orders, is the incremental effect. Lift is 450 ÷ 2,250 = 20%.
Now put money against it. Suppose the ads cost £9,000.00 and the average order is £50.00.
| Platform report | Incrementality test | |
|---|---|---|
| Orders credited | 1,500 | 450 |
| Cost per order | 6.00 | 20.00 |
| Revenue credited | 75,000.00 | 22,500.00 |
| ROAS | 8.33 | 2.50 |
The platform attributed 1,500 orders to the campaign, because 1,500 people who saw an ad later bought. The test says 1,050 of those would have bought regardless. The real cost per order is £20.00 rather than £6.00, and the real ROAS is 2.50 rather than 8.33.
The holdout test page goes through the practical choices: whether to hold out people or whole regions, and how big the groups need to be.
What it can't tell you
It measures one channel, over one period, at one spend level. The 20% lift above says nothing about what a second campaign would add, or what the same campaign would do in December.
It needs enough volume for the difference to be real rather than chance. In the illustration the control group placed 250 orders. With 25, a 20% lift could appear or vanish on luck alone. An account with few orders a month cannot run a clean test on a small channel.
It can't see long-term effects. An ad that builds awareness and pays off in six months looks like it did nothing in a four-week test.
It costs sales. The control group gets no ads, and some of them would have been persuaded.
When to use it
Run one when a channel's budget is big enough that being wrong about it matters and the platform's reported return looks too good to be true. Brand search and retargeting are the usual suspects, because they reach people who already know you.
Don't run one on a channel spending £300 a month. Pause it for a fortnight and see if sales move; that is a crude holdout and it's probably all the question deserves.
If you have many channels and want all of them on one scale, marketing mix modelling is the other route. It needs more data, but it covers every channel at once.
Tracking it in a spreadsheet
The inputs are orders by group, which the test design gives you, plus spend over the test period. Keep the platform's own reported conversions beside them for comparison. A scheduled Fresh Sheets report can keep Google Ads or TikTok Ads cost and conversions, Meta Ads spend, and Shopify orders, current in their own tabs for the test window. Fresh Sheets replaces each tab's contents on every refresh, so keep the test and control counts and the lift arithmetic on a separate tab.
Check whether your test result is significant with the free calculator
Related guides
Marketing attribution: what it answers, why it's hard, and what a small team should do
Marketing attribution is the practice of working out which marketing activity led to each sale, so that the channels that earned it get the credit.Attribution models: six ways to split the credit, and who each one flatters
An attribution model is the rule that decides how much credit each marketing touch gets for a sale.Last click attribution: the default model, and who it flatters
Last click attribution gives all the credit for a sale to the final marketing touch before the order.