Can a small operator run a real incrementality test without a data science team?

Why a holdout beats a better model

Every attribution model, from last click to the most sophisticated multi-touch approach, works on conversions that already happened. It divides observed credit among observed touches. None of them can tell you whether the conversion would have occurred with no touch at all, because that outcome is not in the data.

A holdout creates that missing observation on purpose. You give up some revenue in one area for a defined period in exchange for knowing something you otherwise cannot know.

Choosing the areas

• Similar historical job volume. Comparing your strongest market to your weakest guarantees a confusing result.
• Similar job mix. An area that skews replacement will look different from one that skews repair regardless of what you do to the ads.
• No media spillover. If radio, TV or direct mail crosses both areas, they are not independent.
• Enough volume. If an area produces a handful of jobs a week, one large job will decide your result.

How long, and what to count

Run for at least one full booking cycle plus the tail — if most leads book within three weeks, a two-week test measures noise. Count booked jobs and revenue per area out of the field service system, not conversions out of the ad platform, because the platform's numbers will obviously fall in the area where you turned the platform off.

Also watch the channels you did not touch. If paid is off and organic calls rise, you are watching substitution, and the difference between the two is closer to the true incremental effect than the raw drop.

The honest limits

Low-volume operators may simply not be able to detect a real but modest effect, and that is a legitimate outcome rather than a failure of the test. A null result means "we could not measure it at this size," not "there was no effect." Say it that way when you report it.

Seasonality, weather events and a single unusual commercial job are the most common ways a test gets ruined. Note them while the test runs, not afterward. When we design these with clients we set the measurement plan before the test starts, as part of marketing intelligence work, so the analysis is not invented to fit the result. The reporting side lives in purpose-built dashboards that hold the control and test areas side by side.