Our scanner scored 100%. It was measuring nothing.

How an "accuracy" dashboard with green numbers on it turned out to be measuring coverage, throughput and self-confidence, and what it took to measure whether the models were right.

AIIM runs thirteen things it calls models. Some are classical ML: a random forest for

municipal bond risk, K-means over price shapes, a classifier for muni credit ratings. Some

are statistical: rolling correlations, z-scores on deviations, a G20 inflation and

unemployment forecaster. Some are rules: the nightly technical pattern scanner. A meta-model,`analytics/model_tracker`, watches all of them. It computes drift (PSI and KS tests on

their inputs), proposes improvements, and publishes a scorecard that the app shows on its

Model Dashboard.

For months that scorecard said the market scanner was 100% accurate.

This article is about how that happened, why it isn't unusual, and the pipeline we built to

replace proxy numbers with realized outcomes. The short version: no model's prediction had

ever been compared with what happened next, because almost nothing kept its predictions long enough to compare.

What the scanner "accuracy" actually was

Every model in the tracker reported an `accuracy_score`. None of them compared a prediction with an outcome. Each computed a different stand-in:

Model	What "accuracy score" actually computed
Market Scanner	`min(total_scanned / 2000, 1.0)`: how many symbols it scanned
Predicted Patterns	mean self-reported confidence ÷ 100
Discovered Patterns	number of clusters / 10
Deviation Detection	`1 − deviation_count / 100`
Municipal Bond Risk	`coverage × 0.5 + consistency × 0.5`

The scanner's number is the clearest. It scans about 2,000 symbols a night, so the ratio hit

its cap and stayed there. A scanner that scanned 2,000 symbols and was wrong about every one of them would have scored 100%. It had been pinned at 1.0 since mid-August.

Then the tracker averaged these into an `overall_accuracy`: a plain `np.mean()` of seven

incommensurable numbers (a coverage ratio, a confidence, a cluster count, a throughput), which came to 0.5486. The dashboard rendered that as a headline percentage.

None of this was malicious, and it's worth saying how it happens. Each proxy was a reasonable health check when someone wrote it: "is the scanner covering the universe?" is a fine thing to monitor. The problem was the field name. Once a number lives in a field called `accuracy`, every downstream consumer (the dashboard, the averages, the assistant describing the models to users) treats it as accuracy.

The telemetry was mostly missing too

The audit found more:

• 8 of the 13 models reported "inference_count: 0". Only the bond risk service wrote

  inference logs at all, and it trimmed them to the last 1,000 entries, so there was no

  history.

  
  

• The scanner's evaluation snapshots had never been saved. The tracker serialised them with a raw `json.dumps`, the scanner's payload carried a `NaN` correlation, and Postgres's `json` type rejects NaN. So drift detection could never establish a baseline for the scanner. The fix was a `_json_safe()` pass. The row is still 1.96 MB, because it embeds the whole scan.

  
  

• Twelve of the thirteen models hard-coded "enhancements": []. That's why the

  "recommendations" panel only ever showed one item. Folding the models into a shared `_register_model()` raised that to 19–20.

  
  

• Five tracker inputs read keys that nothing wrote. One of them belonged to an

  on-device P/E model whose code had already been deleted, and the tracker kept reporting it as deployed. Both on-device predictors were later removed from the tracker, and the index, sector, portfolio and G20 entries were repointed at the data their models actually publish.

Step 1: stop calling it accuracy

The first change was a rename, and it was the most important one.

`accuracy_score` became `None` for every model, and the proxies moved to a field called

`health_proxy`. The tracker gained honest counters: how many models have a real accuracy

(none, at that point), how many report any telemetry (5 of 13), and a list of the silent ones.

The D1 history table needed care. A migration moved the old accuracy values into a new

`health_proxy` column rather than copying them. With a copy, old rows would carry the same

number in both columns and be indistinguishable from new rows with a real accuracy. Ordering mattered too: apply the migration before deploying the Worker that inserts into the new column, or every insert fails.

The app needed a guard of its own. It only believes an `accuracy` value when the row or

payload also carries the new `health_proxy` field, so a cached pre-migration payload can't be

re-read as accuracy. We checked that across all three schema states (old, migrated, new)

before shipping.

The assistant in the app (see [Article 8](assistant-in-a-worker)) has a model-tracker

tool, and its description says plainly that these figures are health proxies, not accuracy.

Otherwise a language model would cheerfully tell a user "the scanner is 100% accurate".

Step 2: Find out whether anything kept its predictions

Renaming the proxies was the easy half. To measure accuracy you need two things for every

prediction: what was predicted, as it stood at the time, and what happened afterwards. The

audit went model by model looking for the first half, and almost nothing had it:

• The nightly scan overwrites its Redis key on every run. Yesterday's predicted patterns are gone by tonight.

  
  

• The ML dashboard's index and sector calls are a KV blob replaced hourly.

  
  

• The muni rating scores and bond risk results are upserted on the issuer id. Each

  rescore replaces the last.

  
  

• The county GDP nowcast is discarded when the BEA publishes the real figure, which is exactly when it could have been scored.

  
  

• The G20 forecasts expire from Redis after 24 hours and from KV after three days.

  
  

• A predicted P/E was dropped by the D1 insert; a portfolio's expected return was never

  persisted.

  
  

• A table built for outcomes, `analytics.inference_feedback`, had an `actual` column that

  nothing wrote.

There was exactly one point-in-time store of predictions with outcomes: the intraday

signal log. Every pattern the stream agent emits during market hours is logged with its side

and price, and a 16:45 job labels it with forward returns at +5, +15, +30 and +60 minutes and

at the close. It held 68,800 labelled signals across 26 sessions. Its own docstring said its

results should be published through the model tracker. That had never been done.

What the one honest dataset said

The intraday log's returns are side-adjusted: positive means the signal was right, whatever

its direction. The results were sobering.

• Every pattern hit between 41% and 48% at 30 minutes, with a negative mean return. The worst, a VWAP reclaim pattern, was right 41% of the time and lost 0.19% on average.

  
  

• Confidence was inverted. Signals the detectors were most sure of (confidence 0.9–1.0)

  were right 43.7% of the time; the least confident bucket was right 47.1%. The pattern held every week.

  
  

• Wired into the tracker, the stream's accuracy came out at 46.6% over 41,820 signals

  in the last 20 sessions.

Two details from that analysis are worth passing on.

A rank correlation alone would have missed the inversion. Spearman's rho between

confidence and outcome was only −0.036, which reads as "no relationship". The problem shows up in the gap between the top and bottom buckets, not in a monotonic trend across all of them. The tracker now reports the bucket gap explicitly.

An apparent edge was an artifact of timing. Because the patterns lose money, fading them (taking the opposite side) looked profitable. A walk-forward evaluation found why: signals were being emitted a median of 119 seconds after the bar they described. The "edge" from fading came from entering at a stale price that the market had already moved past. Nobody changed any trading behavior on the back of it, and the logger now records the price at emission time as well as at bar time, so future evaluations measure the trade someone could actually have made.

Step 3: Keep dated snapshots, then label them

The fix for "nothing kept its predictions" is a module, `analytics/prediction_outcomes`,

that keeps a dated copy of each prediction and labels it once its horizon has passed.

Snapshots happen where predictions are made. The scanner snapshot runs at the end of the nightly scan, after enrichment, so it captures exactly what was published. The ML dashboard snapshot reads the payload the Worker is actually serving. The portfolio constructor writes each holding's weight and expected return. Each row records the model, the date, the symbol, the prediction, the side, the model's own confidence and its horizon.

Value forecasts, where the truth isn't a price, are snapshotted too, but only when the

value changes: the muni rating score per issuer (truth: rating actions), the fitted P/E per

country (truth: realized P/E), the G20 CPI and unemployment forecasts at 3, 6 and 12 months

(truth: the official prints), and the county GDP nowcast (truth: the BEA release). Their

labels are separate, because their truth arrives on its own schedule, sometimes a year

later. The point is to start the record now, so it exists by the time the outcome does.

Labelling runs at 22:30 on weekdays. For every price-based snapshot whose horizon has

passed, it computes forward returns at +1, +5, +10 and +20 trading days. Twenty is the longest horizon any daily model claims. Returns follow the intraday log's convention: signed by the prediction's side, so positive always means "the call was right". There's also an `xret_`

version, in excess of SPY over the same window, because a long call on a stock in a rising

market is not skill. The single-stock models are scored on the excess return.

Getting prices was the hard part

The obvious price source, the local `equity_prices` table, covers only about 60–110 of the

roughly 940 symbols a scan returns. So the labeler uses FMP's end-of-day bulk file: one call

per date returns every listed symbol. That endpoint is rate-limited, and FMP's 429 response

warns that repeated calls can get the key restricted. So:

• Each finished trading day is fetched once and cached, and only after 22:00 ET, by which time the close has been final for six hours.

  
  

• Calls are paced at least 20 seconds apart, with a 90-second back-off after a 429; a weekend or holiday file contains only crypto, so "SPY is absent" is the test for "market

  closed".

  
  

• A cached file can't reflect a stock split announced later, so any window spanning a split

  is masked rather than scored.

  
  

• A snapshot that still has no price after 60 days (delisted or renamed) is abandoned rather than retried every night.

Lookahead: The bug that makes everything look good

The most dangerous number in this whole project came from our own end-to-end test. To

verify the pipeline, we ran it inside a transaction that was rolled back afterwards, using

calls that had been re-dated a week into the past. The resulting daily hit rates looked

excellent. They were meaningless: the "predictions" had been made with knowledge of the

prices they were being scored against. We recorded them as lookahead artifacts so nobody

quotes them later, and there is no historical KV data to backfill from, because the scanner

key only ever held the latest scan.

Lookahead also turned up inside a model. The predicted-pattern model's occurrence check had looked at bars beyond the prediction date. It's now causal: "Did this pattern appear within *h* bars after the call?", using only data available at each step.

That model had a second, subtler problem. 80% of its "predictions" were patterns already on the chart when the prediction was made. Predicting that a pattern currently present will

be present tomorrow is mostly a bet on persistence. So the snapshot records an

`already_present` flag, accuracy is reported as precision on new patterns only, and

persistence is reported separately. The production model also turned out to be a Markov

baseline trained on a next-bar target but labelled with a multi-day horizon. Logistic

regression only runs if a flag selects it.

When the model isn't the model you think it is

Measuring things properly exposed models whose labels didn't match their code.

The "Monte-Carlo" G20 forecaster was seasonal-naive. Its forecast bands were centered on the current value, and it had a dataframe indexing bug: for a 12-month forecast, `iloc[-12 + h]` evaluates to `iloc[0]`, the first observation ever recorded. The US policy rate's

12-month forecast was 0.80 when the current value was 3.75. Steps beyond 12 read from the

start of history. The fix was a proper `seasonal_back_index()`. The audit also found stale

data vintages for several countries (one series hadn't updated since 2013), so forecasts now

apply a six-month staleness rule rather than presenting an old vintage as current.

The sector "estimated return" was a trailing change with a forced sign, not a forecast.

It's now described as what it is.

The municipal bond risk model had never seen a real rating. This one deserves its own

section.

The muni model trained on 100 synthetic rows

The Bond Risk screen showed nearly every municipal issuer as high risk. That's an odd result

for a market where most issuers are rated AA or better. The cause was in the training code.

The real-data path needed four fields present on the same document. Not one of the 1,926

rated documents had all four, and a `dropna()` on the labor columns quietly removed every

row. So the model trained on 100 synthetic rows from a hand-written formula whose typical

municipality came out at about 14.9 on a 0–20 scale, which corresponds to a single-B rating.

At inference time the missing features fell back to defaults, and 2,140 issuers received the

identical score of 14.62. Against clean ratings, it was out by 11.2 notches on average,

with an AUC of 0.50: a coin toss.

The labels were dirty as well. The rating normalizer missed upper-cased Moody's ratings

like `AA2` and `BAA1` (more than 900 of them), and counted fragments of PDF text as ratings: a stray "C" from Moody's 496 times, "D", "F" and "E" from S&P. Clean parsing rules produced 1,782 documents across 1,156 issuers, and the real distribution is what you'd expect: 74% AA− or better, 23% single-A, 2.4% BBB.

Choose the Metric for the label distribution

The rebuilt model adds local economic context: each issuer is matched to the most specific

geography available (school district, then county, then census place, then state) and given

local factors. GDP per resident turned out to be the strongest single factor. With local

factors, the rank correlation with actual ratings was 0.23 and the AUC for "below AA−" was

0.65. Using issuer type alone, they were 0.07 and 0.59.

But the rebuilt model's mean absolute error was worse than predicting AA for every issuer: 1.51 notches against 1.38. When three-quarters of your labels sit in one narrow band, a constant prediction has a low MAE and tells you nothing. The question that matters (which issuers are weaker than the rest?) is a ranking question, so rank metrics are the ones to

report. Even then the model shrinks toward the base rate: of 2,320 issuers it placed 1,698 in

Very Low risk, 620 in Low and only 2 in Moderate. The tails are under-predicted, and the app

says so.

In-sample numbers are not results

The next version replaced labels parsed from bond documents with dated rating actions from the agencies' own historical disclosures. Along the way, the step that matched bonds to agency records turned out to be contaminated by stray identifiers inside the documents. That was fixed, and every figure computed on the contaminated matching was discarded.

The figure that survived is the honest one. Evaluated point in time, training only on

labels available as of a cut-off date and scoring out-of-fold on downgrades that happened

afterwards, the old model's AUC was 0.567 and the new one's 0.595. The confidence interval on the difference runs from about −0.03 to +0.07, so it includes zero. The improvement is not statistically significant. Earlier, more flattering backtest figures (0.62–0.65) were

in-sample, and we stopped quoting them. A real forward test needs about twelve months of new rating actions.

That's a disappointing paragraph to write, and it's the most useful one in this article. A

model that might be slightly better, measured honestly, beats a model that looks much better measured carelessly. Only one of those can be improved on purpose.

Where the models run

For completeness, here's the inventory. Several "models" run on the phone or tablet, and not on the server, and only report into the tracker.

Model	Technique	Runs Where	Scored Against
Municipal bond risk	Random forest → classifier on agency labels	Local	Rating actions, months later
G20 macro forecaster	Seasonal baseline (fixed)	Local	Official CPI and unemployment prints
Market scanner	Rule-based technical patterns	Local	Forward returns, excess over SPY
Predicted patterns	Markov baseline (logistic regression optional)	Local	Pattern occurrence within *h* bars
Discovered patterns	K-means on normalized price shapes	Local	Not a prediction; health only
Correlation / deviation	Pearson, z-scores on rolling correlation	Local	Health Only
Index sentiment	Direction calls	Local	5-day forward returns
Sector forecasts	Direction calls	Local	Every horizon (the producer's own horizon is ambiguous)
Portfolio optimizer	Mean-variance	Local	20-day return vs SPY
Intraday signals	Pattern detectors on 1-minute bars	Local	+5 to +60 min, close
On-device pattern detector	SMA, RSI, head and shoulders rules	Device	Not yet scored

The "Scored Against" column is the useful one. Writing it down for each model forces the question "what would it mean for this to be right?". For a few rows the honest answer is that there's no future fact to check: a cluster of price shapes or a rolling correlation describes the

present rather than predicting anything. Those rows deserve health monitoring, not an

accuracy number, and the table says so rather than inventing one.

The sector row shows the other kind of honesty. The sector model's horizon is described as

"30-day" in one place and "next week" in another, so the labeller doesn't pick one: it

scores the calls at every horizon and lets the numbers show where, if anywhere, they work.

A timeline of the fix

The last bars are the honest part. Realized accuracy for a 20-day horizon can't exist until

20 trading days after the first snapshot, and the muni model's forward test needs a year.

Until then the dashboard says "no realized accuracy yet", which is true.

Takeaways

• Name metrics for what they compute. A health proxy in a field called `accuracy` will be

  read as accuracy by every consumer, human or model.

  
  

• Never average incommensurable numbers into a headline.

  
  

• You can't measure a prediction you overwrote. Snapshot predictions at the moment they're published, with side, confidence and horizon.

  
  

• Score against the right baseline: side-signed returns, in excess of the market, and

  precision on new calls, not persistence.

  
  

• Check calibration by bucket. A near-zero rank correlation can hide an inverted top bucket.

  
  

• Hunt lookahead in tests as well as models. The best-looking number is the first one to

  distrust.

  
  

• Pick the metric for the label distribution. With skewed labels, MAE rewards a constant;

  rank metrics don't.

  
  

• Report point-in-time, out-of-sample results with intervals, even when they're

  disappointing. Especially then.