Validation & evidence

Scorer validation status

Updated August 2026. We previously showed a JAK2 spot-check reporting MAE ~0.55 pKd. That table has been removed because the comparison it invited was not a fair one, and we would rather say so than leave it up.

How to read the small tables above. Both are marketed-inhibitor spot-checks at n=8 and n=9. At that size a rank correlation is barely distinguishable from noise, and marketed drugs of one class all sit in a narrow potency band — which is exactly the situation where predicting a constant wins. They are published for transparency, but they are too small to settle anything. The larger test below is what we now rely on.

External validation on unseen structures

The decisive question is whether the scorer works on proteins it has never encountered. We took 516 PDB complexes with measured binding affinities, released between 2022-06-08 and 2024-10-23 — after the PDBBind 2020 data the model was trained on, so none can have been in training. Measured pK_d spans 1.05–11.00, so a constant prediction cannot win by default.

The recalibrated row is what the API now serves. Broken out by measurement type, it beats the constant baseline on each independently:

K_d carries the least assay noise and shows much the strongest correlation (ρ = 0.63 on n = 65); K_i and IC₅₀ are assay-dependent, so some of their error sits in the labels rather than the model. The gap between 0.63 on clean K_d and 0.34 on K_i is a reasonable indication of how much label noise is costing us. The calibration coefficients were fitted on the external set and are the ones the API ships; they were not refitted after the pocket correction, so the improvement above comes from the inputs rather than from a retuned output layer.

What it can actually do: enrichment

The honest performance claim is enrichment on a diverse library, not absolute accuracy. Across the 516 external complexes:

In practice: screening the predicted top fifth of a diverse library surfaces about 1.7× as many potent compounds as choosing at random, and tightening to the top decile roughly doubles the hit rate against base rate. That is a modest, real advantage for first-pass triage — and it is the strongest claim the evidence supports. It is not a substitute for docking, FEP, or an assay.

Three things this establishes. First, the previously reported 0.2046 pK_d internal figure was a product of leakage: raw external MAE is roughly seven times worse. The training split was randomised per complex rather than per target, leaving a validation set dominated by near neighbours of training examples (median nearest-train pocket identity 67.8%; 15.8% with a ≥90%-identical training pocket). Second, the model carries real transferable signal — ρ = 0.39 across 516 unseen complexes is still weak, but at that sample size it is unambiguously not zero, and on clean K_d labels it reaches 0.63. Third, we previously wrote that the dominant defect was calibration, citing a raw head sitting 1.43 pK_d low. That was mostly an artefact of the undersized pockets. Fed the input it was built for, the raw head sits only 0.28 pK_d low, and an offset correction buys almost nothing (MAE 1.353 → 1.344). The remaining gain from recalibration is variance shrinkage, not offset removal.

Why the served predictions have a narrow spread. The fitted slope is 0.279, so calibrated values cluster between about 5.7 and 8.0 rather than spanning 1.8–10.1 as the raw head does. This is deliberate and it is what a correctly calibrated weak predictor looks like: at ρ ≈ 0.4, hedging toward the mean is the accuracy-optimal response, and the wide raw spread was overconfidence rather than information. Rank order is essentially unaffected — the mapping is monotone, and 508 of 516 compounds hold the same position — so library triage behaves identically. The uncalibrated score remains available as raw_affinity in the API for anyone who wants the wider relative spread.

A correction to an earlier version of this page. We previously stated that raw predictions were roughly 2.6× compressed relative to reality. On unseen structures that was false: the raw head's spread (SD 1.63) matched label spread (SD 1.61) almost exactly. The apparent compression was an artefact of testing only on marketed drugs, which occupy a narrow potency range by construction. We are recording the error rather than quietly deleting it.

What the scorer is, stated plainly. A first-pass enrichment filter, roughly 1.7× better than random on a diverse library at the top quintile. It separates broad potency regimes; it does not order compounds within one. Use it to shorten a long list before docking. Do not use it to choose between two similar compounds, do not use it for lead optimisation within a series, and do not read the absolute pK_d as a predicted K_i. The August 2026 correction moved the ranking metrics up; it did not change what the tool is for.

What this does not affect. Deterministic workbench signals are unchanged and independently verifiable: RDKit physicochemical properties (MW, LogP, TPSA, HBD/HBA, QED, Lipinski), ChEMBL similarity and clinical-analog flags, AutoDock Vina docking poses, and the 3D pocket viewer. None of these depend on the affinity model.

Reproducing this. The external set is every single-protein PDB entry carrying an rcsb_binding_affinity record released after 2022-06-01 and absent from our training corpus, queried through the RCSB search API. Pockets are whole residues having any atom within 6.0 Å of any ligand atom, which reproduces the training set's median of 34 residues per pocket. Structures, pockets, labels and per-complex predictions are available on request — including the cases we get wrong, and including the superseded 6.5 Å-centroid predictions behind the older numbers.

External spot-check: known EGFR inhibitors

Same scorer, different target. Literature pKd from ChEMBL kinase assays (rounded). Scored live against EGFR pocket 2ITK, June 2026. Viewer may show PDB 1M17. Use for transparency, not as a global MAE claim.

Generated libraries (Pathfinder exports)

These tables show RL-optimized candidates ranked by the same Stage 6 scorer. RL steps (200 vs 500 here) control how long generation searches — not how the affinity model was trained. Higher top pKd after 500 steps partly reflects more optimization time, not a different model. Do not compare raw top pKd across runs with different step counts as proof one target is “better.”

EGFR library · 200 RL steps

Default Pathfinder demo — 10 compounds from a 200-step generation run, batch-scored and ranked by predicted pKd on pocket 2ITK.

GEN-12 ChEMBL top hit: CHEMBL551595 (80% Tanimoto).

Spotlight: GEN-12 · EGFR hit #6

Example of orthogonal signals med chem teams expect — not a single opaque score.

JAK2 library · 500 RL steps

Separate 500-step generation run — same scorer, more RL optimization time. Top pKd ~9.25 vs ~8.18 on EGFR (200 steps) is not apples-to-apples; use external spot-checks above for cross-target comparison.

Top hit GEN-1 ChEMBL analog: CHEMBL175030 (100% Tanimoto). Scoring pocket: PDB 2b7a.

Within-library vs external benchmark

External spot-checks test the scorer on approved drugs. Generated libraries test rank-ordering within your series on a fixed pocket. A higher top pKd after 500 RL steps does not mean one target is better than another; it means generation had more steps to climb the same scorer — so cross-run pKd comparisons are not meaningful.

Where it does not work: screening libraries

Every number above is measured on protein–ligand co-crystal structures. Those are, by construction, compounds that bind something. A screening library is not: it is roughly 99% non-binders. Those are different problems, and this page previously implied one result covered both. It does not.

Following the argument in Ahmed et al. (JCIM 2026) — that benchmarks built from curated ligand sets inflate apparent performance — the scorer was run against four real primary screens from PubChem BioAssay. 30,000 compounds each, randomly sampled so the assay's true hit rate is preserved rather than enriched. No compound failed to score.

The two enrichment figures above 1.0 are not real. Thrombin's 3.23× is two compounds where chance predicts 0.6; HSP90's 1.81× is five where chance predicts 2.8. Neither is significant on its own, and pooled across all four targets the top 1% contains 10 actives against 7.8 expected (P = 0.26). Mean ROC AUC across 120,000 scored compounds is 0.499, which is the value for random ranking.

Why, and what it still does

The model is trained on PDBbind: protein–ligand complexes that have been solved. Every training example is therefore a compound that binds. It has learned to separate strong binders from weak binders and has no representation of a non-binder. Directly measured: across 1,612 diverse screening compounds its entire output range is 0.92 pK_d, with 80% falling inside 0.38 pK_d of one another. Rosiglitazone and pioglitazone, both marketed PPARγ drugs, score 7.00 and 6.99 against PPARγ — against a random-compound mean of 7.03 on the same target.

The co-crystal results remain valid for what they measure: ordering compounds that already bind. A ligand-scrambling control — scoring each pocket with a different complex's ligand — leaves ρ = 0.11 (mean of three seeds), so roughly 72% of the published ρ = 0.39 is genuine ligand-level signal rather than a per-target prior.

Read the claim narrowly. This is useful for ranking compounds that are already plausible binders and that span a wide potency range. It is not a hit-finding tool, and nothing on this site should be read as saying otherwise.

A separate model for screening

Everything above describes the affinity scorer behind /score, and it remains true of it. The diagnosis — that the model had never seen a chemical non-binder — suggested a fix: train the binder head on real primary-screen outcomes rather than on PDBbind. That was done, holding out one target at a time across the four assays, and it works, modestly.

Improved on all four targets, significant on three. For scale, published methods on LIT-PCBA reach ROC AUC 0.53–0.61 (Glide-SP 0.5315, DrugCLIP 0.5717, GNINA 0.6093), so this is competitive with the field rather than ahead of it. It is a weak signal in absolute terms and should be read as triage, not as hit-finding.

Why it is a second model rather than an upgrade. The gain comes from training the shared interaction layer, which is also what the affinity head reads from. Measured pK_d drift for those weights is 0.947 to 4.268, against a real post-cutoff MAE of 1.222 — folding them in would destroy absolute accuracy and the Vina result below. Restricting training to the binder head alone avoids that completely (drift 0.000, verified) but learns nothing: mean AUC 0.5179 to 0.5175. So the two run side by side, and the screening model's affinity head is never read.

A note on how these were measured

Splitting by target is not sufficient on PubChem screens, because they reuse compound libraries: 30–42% of each held-out set also appeared in training as a row for a different target. The effect is large. A parameter-free ECFP4 nearest-neighbour baseline scored EF1% 5.27× on the JAK2 holdout and 2.51× on thrombin; after removing the shared compounds those fall to 0.91× and 0.33×. The entire apparent enrichment was the same molecule matching itself. Every figure in this section is measured after de-overlapping. This is the same class of defect an independent audit documents in LIT-PCBA, which is worth knowing before comparing against numbers published on it.

Head-to-head against AutoDock Vina

A correlation coefficient on its own means little. This compares the scorer against a reference every computational chemist already knows, on the same 513 protein–ligand complexes, every structure deposited after the model's training cutoff. Both predictors saw identical inputs; the comparison is paired per compound.

The result is a tie, and it is reported as a tie. A paired bootstrap over 2,000 resamples puts the difference at +0.011 in Vina's favour, 95% CI [−0.076, +0.094]. That interval contains zero and narrowed as the sample grew (it was [−0.107, +0.115] at n=259), so this is a genuine draw rather than a sample too small to separate them. VectaBind is not better at rank correlation. It is better at top-decile enrichment — 51% of its top tenth are potent against a 23% base rate, versus 37% for Vina — and it produces that ranking in about 0.46 s per compound against roughly 58 s for docking, without needing a dockable receptor. (Enrichment here uses the same pK_d ≥ 8 threshold as the benchmark section above, so the two are directly comparable; the 2.25× and the 2.23× reported earlier differ only because three complexes failed to dock.)

Two controls, because the headline is worthless without them

1. Redocking. Before asking whether docking predicts potency, it has to reproduce poses that are already known. Each crystal ligand was docked back into its own receptor and compared with where it actually sat: median RMSD 0.99 Å across 507 structures, 76% within 2.0 Å. The field's usual bar is 50%. An earlier attempt that docked into carved pockets rather than deposited structures failed this control badly — poses landed up to 16.8 Å from the site — and every number from it was discarded.

2. Search effort. Vina ran at exhaustiveness 8, its own default but a modest setting, so 57 complexes were re-docked at 32 — four times the effort. Rank correlation moved from +0.376 to +0.404, the two settings agreed at ρ = 0.995, and the median score difference was 0.04 kcal/mol. The threshold for "this changes the conclusion" was fixed at 0.05 before the check was run. Vina was not handicapped by the default.

Method and failures. Receptors were the deposited PDB entries with waters and crystallisation additives removed and cofactors retained; the search box was centred on the crystal ligand and that ligand stripped before docking. 12 of 1,032 jobs failed (1.2%) — 9 redocking controls and 3 compound docks — from no-pose returns, subprocess timeouts, and two ligand-naming mismatches. Failed jobs are excluded from the counts above rather than silently reducing the denominator. Model predictions used the same pockets and calibration as the shipping API.

Signal provenance

Every column in the workbench traces to a named source — not a black-box number.

Model benchmark (secondary)

Internal PDBBind 2020 validation MAE is reported in Methods. For screening workflows, rank order within your library and ChEMBL-grounded analog checks are more actionable than a single global MAE.