Summary. In a simulated self-driving lab over 2,311 PTP1B inhibitors from ChEMBL, calibrating the scoring model with isotonic regression left hit recovery unchanged at a 360-experiment budget (35.5 ± 3.5 versus 35.3 ± 4.2 hits over 8 seeds) and hurt single-objective selection. Isotonic regression collapsed 1,844 distinct scores into 74 levels and placed 100 molecules at exactly 1.000, so the top of the ranking, where each batch is chosen, became a tie.
The question
A closed-loop discovery system is a control loop. A model scores a pool of candidates, the system tests a batch, and the results retrain the model. The common intuition is that better-calibrated probabilities should make this loop more efficient, since the acquisition step consumes those probabilities as if they were true.
I tested that intuition directly: run the same loop with and without isotonic calibration of the scoring model, and count how many actives each recovers within a fixed experimental budget.
What happened
| Scoring | Hits at 360 experiments | Distinct scores |
|---|---|---|
| Raw model scores | 35.3 ± 4.2 | 1,844 |
| Isotonic-calibrated | 35.5 ± 3.5 | 74 (100 tied at 1.000) |
The difference in hits is well inside seed-to-seed variation, so calibration bought nothing on average. In the single-objective setting it did worse. The more informative number is the last column.
Why: calibration can trade away resolution
Isotonic regression fits the monotone function that best maps scores to observed outcomes:
f* = argminf non-decreasing ∑i (yi − f(si))2
On finite data the solution, found by the pool-adjacent-violators algorithm, is a step function. Adjacent candidates whose labels disagree with their order are pooled into a single level. That never reverses two candidates, but it can merge them, and merged candidates cannot be ranked against each other.
The acquisition step only reads the top of the ranking. When 100 molecules share the maximum score and the batch is smaller than that, the model's remaining ordering information is discarded at the one place it mattered. Calibration improved how well the scores matched outcome frequencies, while destroying the fine ordering the decision depended on.
Why it matters beyond this experiment
The failure is general: a pipeline that calibrates and then ranks can lose discrimination at the decision boundary while every calibration metric improves. The same pattern applies wherever confidence scores are post-processed before a top-k or thresholded decision, in perception as much as in discovery. The metric that matters is the one the decision consumes.
Remedies worth testing
- Break ties with the raw score. Rank by calibrated probability first and raw score second, keeping calibration where it is useful without discarding order.
- Use strictly monotone calibrators. Platt scaling and temperature scaling are strictly increasing, so they preserve the full ranking while still correcting overconfidence.
- Evaluate the decision, not the probability. Report hits at budget and enrichment at the top of the ranking alongside calibration error, since the two can move in opposite directions.
Limitations
This is a preliminary result: one target (PTP1B), one dataset source (ChEMBL), and 8 random seeds. The no-gain result needs confirmation across more targets and scoring models before it generalizes, and the remedies above are hypotheses, not yet tested.