Advertising holdouts and ABM: preserve the counterfactual

Ask the counterfactual question

An account that sees an advertisement may already be more likely to buy. Platform delivery can depend on activity, bidding and predicted response. Comparing exposed accounts with all unexposed accounts can therefore compare different starting conditions.

A causal question asks what would happen to comparable eligible accounts without the intervention. Define the intervention narrowly: one campaign, one added sequence or one coordinated account play. Record other sales and marketing work that continues in both conditions.

What ghost-ad research contributes

Johnson, Lewis and Nubbemeyer developed ghost ads to identify control counterparts of exposed users in randomised advertising experiments. Their 2017 Journal of Marketing Research paper explains how counterfactual exposure records can improve measurement and the economics of testing.

This method relies on platform support. An ordinary CRM export cannot recreate hidden auction allocation. Our ABM application is the broader experimental principle: preserve who was eligible and assigned before comparing people who happened to receive or engage with an ad.

Explore the original methods and findings in Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness.

The practical workflow

1. Specify the intervention
2. Freeze eligibility and assignment
3. Monitor account spillover
4. Count defined outcomes
5. Report uncertainty and delivery

Compare the approaches

Define the account as the experimental unit

If several stakeholders belong to one buying group, assigning their contacts independently can allow the same account to receive both conditions. Consider account-level assignment when the intervention coordinates multiple contacts. Record parent and subsidiary relationships that could create spillover.

Set exclusions before observing results. Keep assignment, delivery and engagement as separate fields. Analyse the assigned groups as planned, while reporting implementation failures. Excluding every unengaged account after the campaign can change the question and introduce selection bias.

Work through a fictional holdout

Suppose 200 eligible accounts are assigned equally to the ordinary programme or the ordinary programme plus a new advertising play. During the observation window, 12 accounts in the added-play group and 9 in the baseline group reach the defined outcome. Those are illustrative counts, not evidence of a reliable uplift.

Report the group sizes, the three-outcome difference and uncertainty. Check whether seller activity, opportunity maturity and account spillover remained comparable. Do not claim success solely because the point estimate is positive. A small test may reveal delivery problems before it can estimate a commercial effect precisely.

Keep response and treatment effect distinct

Treatment-effect methods, including the metalearners studied by Künzel and colleagues, address how an intervention’s effect may vary with context. They still require defensible identification assumptions and adequate data.

A propensity score that predicts likely conversion answers a different question. Highly likely buyers may convert with or without advertising. Begin with a valid experimental design before seeking subgroup effects. Sparse enterprise outcomes can make elaborate subgroup claims especially unstable.

Explore the original methods and findings in Metalearners for estimating heterogeneous treatment effects using machine learning.

Choose a realistic learning objective

For a small account universe, operational outcomes such as accepted handoffs can be measured sooner than closed revenue. Specify both the short-term learning objective and the longer commercial observation window. Avoid replacing the commercial outcome after results arrive.

Keep the protocol, assignment record and exclusions with the report. If the test cannot distinguish a meaningful commercial effect, say so and use its operational findings to improve the next evaluation.

For the next part of this decision, read Run ABM Experiments With Few Independent Accounts.

Your next-action checklist

• Assigned groups: Preserve the original assignment. Check the limitation: needs adequate independent accounts.
• Exposed groups: Separate from causal analysis. Check the limitation: exposure may be selected.
• Attribution: Label as contribution evidence. Check the limitation: does not supply a counterfactual.
• Subgroup effects: Predefine and validate the analysis. Check the limitation: sparse data can overfit.

Use the comparison to choose a bounded next step. Record the evidence, the responsible owner, and the review decision before extending the play to additional accounts.

Questions this guide answers