Measuring what works

Holdout test: the simplest way to measure what your ads add

A holdout test withholds your ads from one group of people, keeps showing them to another, and compares what the two groups bought.

About the Shopify connectorAbout the Google Ads connectorAbout the Meta Ads connectorAbout the TikTok Ads connector

The difference is what the ads added. It is the simplest design for incrementality testing.

Why it matters

Platform-reported conversions include people who would have bought anyway. A holdout measures how many. The decision it changes is whether a channel earns its budget at the level you spend. An attribution report can't answer that, because it only divides credit among touches that were present. A holdout measures what happens when the touch is absent.

How it works

There are two ways to pick the group that goes without.

An audience holdout splits people. The platform withholds the ad from a random slice of the audience it would otherwise reach, and reports purchases from both slices. Random assignment makes the groups alike, which is its strength. It depends on the platform being able to track purchases in both groups.

A geo holdout splits places. You turn the ads off in some regions and leave them on in others, then compare sales by region from your own order data. Regions differ in ways you can't randomise away, so it is cruder. Its advantage is that it uses your own sales figures and doesn't rely on the platform's tracking at all.

Here is a geo holdout illustration with made-up numbers. Two regions with similar sales history. Ads stay on in region A and are switched off in region B for 28 days. The 28 days before the test are the baseline.

Both regions rose, because the test ran into a busier month. Region B rose by 100 with no ads, so that much of A's rise is the season rather than the ads. Without ads, A would have reached about 3,500 + 100 = 3,600. It reached 4,200. The incremental orders are 4,200 - 3,600 = 600.

Put spend against it. Suppose the ads in region A cost £12,000.00 over the 28 days. The incremental cost per order is 12,000 ÷ 600 = £20.00. If the platform reported 1,200 conversions for the same campaign, its cost per order was 12,000 ÷ 1,200 = £10.00, and half of what it claimed would have happened anyway.

What it can't tell you

Sample size decides whether the result means anything. The difference you are looking for has to be larger than the normal wobble in the group's sales from one period to the next. If region B's orders usually swing by 300 between 28-day periods, a measured gap of 600 is probably real; a gap of 150 is probably noise. The way to know your wobble is to look at past periods with nothing changing and see how much they moved. For a sense of scale, Meta's own conversion lift studies, which split people rather than regions, will not show a result at all until the test and control groups have at least 100 conversions between them (Meta Business Help Centre, Conversion Lift), and that is the floor for displaying a figure, not for trusting one. The honest version is that small groups with few orders can only detect large effects.

A geo holdout can be confounded. A regional promotion, a competitor opening a store or a local news story lands on one region and not the other, and the test reads it as the ads. Pairing regions with matching history reduces this and doesn't remove it.

It measures one spend level. Switching a channel off entirely tells you what the whole budget is worth, and says nothing about what the last £1,000 of it is worth. The diminishing returns page covers why those differ.

It also costs the sales the holdout group would have made.

When to use it

Run one when a channel's budget is large enough that a wrong answer costs more than the test does, and when the channel's reported return looks suspiciously good. Brand search and retargeting are the common candidates, because they reach people who already know you.

Prefer an audience holdout when the platform offers one and your conversions happen online, where the platform can count them. Prefer a geo holdout when you want the answer from your own order data, or when the purchase happens somewhere the platform can't see.

Don't bother for a channel spending a few hundred pounds a month. Pause it and watch. If you can't see the difference in your sales, the channel is too small to measure, and probably too small to matter.

Tracking it in a spreadsheet

The inputs are orders by region (or by group) for the baseline and test periods, and spend over the test. A scheduled Fresh Sheets report can keep Shopify orders and Google Ads, Meta Ads or TikTok Ads spend current in their own tabs. Fresh Sheets replaces each tab's contents on every refresh, so keep the before-and-after table and the lift arithmetic on a separate tab.