Failure modesSeven, namedFixes where they exist
Seven things I know are wrong with this model.
Or with how I reported it. Fixes and conditions for each are in known_failure_modes.md. Where no fix is applied, it says so.
Zero-impression pages cannot decline, and they carried the result.
11.3% of rows were guaranteed negatives; removing them dropped ROC AUC from 0.750 to 0.565.
The committed report did not reproduce.
Published Precision@50 of 0.680; a fresh clone returns 0.740. The 0.680 is withdrawn.
The 90-day query window opens inside the label month.
window_start is 2026-04-02, one day after the decision date. Excluded on measurement, not assumption.
GA4 features are 59% missing, and missing for the wrong reason.
If importance concentrates there, the model is identifying clients rather than measuring engagement.
The label is defined differently in the two slices.
The CSV's trend_direction is built from impressions; the warehouse label is built from clicks. Numbers from one do not transfer to the other.
Precision@50 is the most fragile metric in the table.
It reads only the top 50 rows, so it moves first when tie-breaking changes. Quote ROC AUC alongside it, never alone. The pipeline now selects the best model by ROC AUC rather than Precision@50.
On the valid population, no model discriminates well.
Best ROC AUC across three models is 0.583. The decision tree returns 0.505, indistinguishable from random, and the hand rule 0.380, below it.
No fix applied — this is a finding, not a defect. The next step is a feature question, not a model question.
This describes the reference pipeline's single-runtime measurement on 26,612 rows; the paper's cross-validated run is a separate measurement and reports higher figures.
where it's written up
Items 1, 2, and 4 have the full working.
The evidence behind them is on the cases page, with the code that produced every figure.