Lead-score calibration: when 80 does not mean 80%

Name what the score represents

An account score may combine fit, activity and recency to order a queue. That does not make a score of 80 an 80% chance of a meeting. A probability requires a defined event, population and observation window.

Start by asking whether the score is intended for ranking or forecasting. A seller with capacity for twenty accounts may need a useful order. A team planning capacity may need a reliable estimate of how many qualified outcomes to expect. Avoid using the same unlabeled number for both tasks.

What calibration research found

Guo and colleagues studied confidence calibration in neural networks using image and document classification datasets. Their ICML 2017 paper found that predicted confidence could differ from actual correctness, and evaluated post-processing approaches including temperature scaling.

The research does not validate a particular ABM model. Our application is to inspect whether a model’s numerical confidence corresponds to observed account outcomes. A model can rank accounts reasonably while overstating the probability assigned to its top group.

Explore the original methods and findings in On Calibration of Modern Neural Networks.

The practical workflow

1. Define score versus probability
2. Specify outcome and window
3. Freeze features before the outcome
4. Review bands and ranking
5. Monitor later cohorts

Compare the approaches

Define the event and evaluation cohort

Choose one event, such as an attended qualified meeting within a stated window. Write qualification criteria and exclusions before evaluation. Use a later cohort for validation, preserving the account state that existed when the prediction was made.

Do not include future seller notes or opportunity stages in the original features. Those fields can leak the outcome into the predictor. Track accounts whose observation window is incomplete separately, rather than classifying them prematurely as failed conversions.

Inspect an illustrative probability band

Suppose a fictional model assigns thirty accounts an average predicted meeting probability of 0.8. If only twelve reach the defined event after the full window, the observed rate is 0.4. This example suggests a calibration problem in that band, but thirty accounts are a limited sample.

Review several bands and report their sizes. Check whether the population changed or the outcome definition was applied inconsistently. If the score was never designed as a probability, remove the percentage interpretation and evaluate ranking at the team’s actual capacity instead.

Report ranking alongside reliability

Precision–recall research helps explain why useful positive predictions matter when outcomes are uncommon. For sales, inspect how many relevant outcomes appear among the accounts the team can actually work. Keep the event prevalence with that report.

Calibration, ranking and operating value are different dimensions. A calibrated model may offer little prioritisation value if every account gets the same probability. A strong ranking may still need recalibration before it informs capacity forecasts.

Explore the original methods and findings in The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.

Use scores as inspectable routing inputs

Show the reason for priority, the supporting signals and the unresolved questions. Set thresholds according to capacity and the cost of missing or incorrectly prioritising an account. Record overrides with reasons so they can be studied.

Recheck calibration on later cohorts and after material changes in channels, qualification or account mix. Route stale or incomplete evidence to research rather than letting an impressive numerical score conceal it.

For the next part of this decision, read Designing an intent score that explains a prioritization decision.

Your next-action checklist

• Ranking score: Evaluate at available capacity. Check the limitation: does not imply probability.
• Calibrated probability: Compare bands with later outcomes. Check the limitation: depends on event and population.
• Threshold: Document the routing rule. Check the limitation: capacity and costs matter.
• Override: Record the reason and result. Check the limitation: can introduce selection.

Use the comparison to choose a bounded next step. Record the evidence, the responsible owner, and the review decision before extending the play to additional accounts.

Questions this guide answers