In the second quarter of 2024 we ran a prospecting campaign that the ad platform reported at 4.2 return on ad spend. Finance reported the same quarter as down. Both numbers were correct, which is the uncomfortable part.

What the platform was counting

The campaign targeted people who had visited the site in the previous thirty days. The ad platform counted a conversion whenever someone who saw an ad bought within seven days. Most of those people were going to buy anyway. The campaign was billing us for demand we already had.

This is not fraud and it is not a bug. It is the platform answering the question it was asked. The question was wrong.

The test that settled it

We ran a geographic holdout: eleven regions with the campaign paused, matched against eleven where it kept running, over six weeks. The paused regions lost about four per cent of revenue. The platform had been claiming credit for roughly thirty per cent.

Six weeks of a slightly smaller business bought us a number we could trust. That trade is almost always worth making, and almost nobody makes it.

Three habits that came out of it

Hold out before you scale. Any channel above ten per cent of spend gets a holdout at least once a year. The cost is a small dip. The alternative is scaling something that does nothing.

Separate new customers from all customers. Prospecting campaigns are judged on new customer revenue only. This one change turned three of our five campaigns negative on paper, and all three were negative in reality.

Write the expected result before the test. If you cannot say beforehand what number would make you stop the campaign, you will not stop it. You will find a reason the number was unusual.

What we teach with it

The full dataset from this campaign, anonymised, is in the marketing path. Students see the platform report first and are asked to decide whether to increase the budget. Most say yes, which is the correct answer given the information available and the wrong answer given reality.

Then they get the holdout data. It is the most useful forty minutes in the course, and it is entirely because the campaign failed.