Which pages should you refresh first? An honest, validated answer.

Abstract

Introduction

A content team with a few thousand published pages cannot review all of them by hand every sprint. Someone has to decide what gets opened first, and the two ways of being wrong do not cost the same. A page flagged wrongly wastes an editor's time. The mistake nobody notices only surfaces when the client asks — and that one is fatal.

The model can fake a good reading. Given a random split it learns the client roster instead of the signal, and the score looks strong right up until it meets an account it has never seen.

This project asked whether a model trained on the FlyRank content extract can rank that queue better than the obvious rule an editor would write without one — and, just as importantly, whether it can say so honestly: with the validation design that actually earns the number, and the limitations stated before a reader has to find them.

Data

Two data sources were used across this project, for two different jobs, and only one of them produced the numbers below.

• The anonymized starter extract — data/raw/content_refresh_anonymized.csv, 30,000 rows, one row per content item. This is the source for every number on this page: the model, the baseline rule, the leakage audit, and the ranked queue. The label, trend_direction, is down when 30-day impressions fell against the prior 30 days.
• The FlyRank warehouse (queried in earlier weeks of this project) — used to develop and stress-test the leakage-hunting method on a genuine past-to-future label (features from one calendar month, label from the next), not to produce this page's headline numbers. Its label is defined on clicks, the extract's on impressions; the two are not comparable and nothing here mixes them.

Excluded from the scored population, and why: the label and its raw material (never used as features); 1,205 rows with no recorded search position; and 2,191 rows where prior-period impressions were zero, meaning decline is arithmetically impossible for that row rather than a real observation of stability. No client names, URLs, or raw search queries appear anywhere in this dataset — client_id/content_id are opaque hashes.

Methodology

Label and baseline

The label is a same-extract comparison, not a forecast — both halves of the 30-day-vs-prior window sit inside the same 90-day pull. The baseline a human would use without a model: a page is worth reviewing if it earns a smaller share of the clicks its position tier normally delivers, where "normally delivers" is measured per tier (weighted clicks/impressions), not assumed, and only scored once expected clicks clear a floor of 5 — below that floor a real zero-click page and a real underperformer are statistically indistinguishable.

Features and validation design

Two feature sets were built and reused everywhere: permissive (38 columns — everything that is not the label, an identifier, or a rule-derivation column) and strict (34 columns — permissive minus four columns that share a time window with the label's own impression comparison, even though they are not the label itself).

The population spans 31 clients, and one client alone is 26.2% of the rows — a random train/test split would let a model learn which client this is instead of the actual signal. Every result below uses a client-grouped 5-fold split (no client appears in both train and test within a fold) unless stated otherwise.

The leakage audit

Five checks, run against the shipped (permissive) feature set:

Results

Same 5 client-grouped folds for every row in the table below:

ROC AUC and precision-at-K do not always agree: RandomForest/strict reads slightly higher at the top of the queue (P@50 0.868) than permissive (0.844), even though its overall AUC is lower (0.611 vs 0.645). Both numbers are reported because picking whichever one looks better would be fitting the story to the metric, not the other way around.

Limitations

• Client memorization is real and mostly what separates the two feature sets. The +0.122 grouped-vs-random gap means a real share of the model's apparent skill is recognizing which client a row belongs to, not predicting decline. For a client not in this 31-client population, only something closer to the non-memorized share of that skill should be assumed to carry over.
• Not clean of same-window correlation. The window-overlap columns carry +0.034 AUC on their own. At the row level, the average permissive-vs-strict probability gap is small (+0.004), but 15.3% of rows (4,083) shift by 0.05 or more between the two feature sets — the AUC cost concentrates in a minority of rows rather than spreading thinly across all of them.
• One fold is one client, not an average. The smallest of the 5 grouped folds is a single client (6,981 rows) — read its number as one client's result, not a representative average.
• Missing data is structured, not random. word_count is missing on 25.7% of the raw extract, concentrated in one client (81.8% missing) and in low-ranking pages (49.3% missing at the lowest tier vs. 11.5% at the highest) — any test on the non-missing rows is measuring a sample biased toward already-better-performing content.
• days_since_last_update cannot support a staleness claim. 57 distinct values cover 30,000 rows, five of which cover 87.8% of them — batch timestamps, not per-page freshness. Nothing here uses it as a freshness signal.
• Cross-sectional, not causal, not a forecast. Every number describes one 90-day extract. “The model ranks pages by measured decline probability” is defensible; “refreshing these pages will fix decline” is not — no intervention was run here, only an association was measured.
• Two datasets, two labels, not comparable. The warehouse leakage-hunt used a clicks-based, past-to-future label on a different, larger population; nothing on this page averages or compares its numbers directly against the extract-based results above.

Reconciling this page against earlier published numbers

Two populations from the same 30,000-row extract have been published from this repo, under two filter orders. Both are correct. Neither previously stated which pipeline produced it, so the figures read as if they disagree.

The row counts differ by 8 because the two filters overlap. 1,205 rows have no recorded position (avg_position is stored as 0, not as a missing value) and 3,388 have zero prior impressions; 1,197 rows are in both, so 3,396 rows fall to one filter or the other. Applying only the impression filter leaves 26,612; applying both leaves 26,604. The 2,191 figure quoted in the Data section above is the incremental removal after the position filter has already taken 1,197 of them.

Ranked recommendations

The same out-of-fold model scores and rule scores above sort every page into one of four tiers: rule and model in agreement rank highest; rows where the rule is silent but the model is confident are flagged for investigation rather than automatic action; rows the rule flags but the model doubts stay in the queue, because the rule and the model answer different questions (is this page under-clicking its position, versus is it trending down).

Reproducibility

• Seed42, fixed throughout
• EnvironmentPython 3.12, scikit-learn 1.9, pandas 3.0
• Capstone notebookwork/notebooks/capstone.ipynb
• Full notebook trailwork/notebooks/ (w01 through w07, the weekly record this page synthesizes)
• Repositorygithub.com/Jericho-Ram/FlyRank-Internship-ML

To rerun: clone the repo, pip install -r requirements.txt, open work/notebooks/capstone.ipynb, Run All. Every number and chart on this page regenerates from data/raw/content_refresh_anonymized.csv with no external dependency or hidden state.

Acknowledgments & data credit

Built on the FlyRank ML Internship dataset, an anonymized content-performance extract provided for this internship. This page is the ML-11 capstone of AI n-able Yourself — the paper and the notebooks it mirrors, not a tutorial on method. Section 3's methodology critique in the underlying notebook responds constructively to FlyRank's published SEO research paper (March 2026, included in this repo under docs/).