Anatomy of a Nightly Market Scan
One Prefect flow produces the dataset behind five screens. Here's every stage it runs, what ends up in a scan row, where it goes afterwards, and how to work out why it looks stale.
Most of what the AIIM app shows about individual stocks comes from one dataset: the nightly market scan. The Pro Scanner grid, the Signals screen, the Dashboard's pattern breadth, the Industry rollup's technical votes, the portfolio links and the assistant's signal tools all read it. It's produced by one flow, `orchestration/market_scanner_flow`, deployed as `daily-market-scan` on a `0 21 * * *` cron in New York time.
21:00 is late on purpose. The close is at 16:00, the post-close pricing update runs at 16:30, and the day's end-of-day bars are settled by evening. Scanning at 21:00 means every pattern is evaluated on a complete daily bar, and the run (which takes a couple of hours once enrichment is included) finishes well before anyone opens the app the next morning. It runs every day, weekends included; on a non-trading day it simply rescans the last session's bars for a different set of symbols.
This article follows one night's run from start to finish, then looks at the parts that cause
trouble: three different writers for the same key, an "as of" timestamp with several names, and fields that look rich in the schema but carry almost no information.
The Market Scan Flow at a Glance
The stages run in sequence, and their failure policies differ deliberately. The scan itself is
fatal: it retries twice, five minutes apart, and if it still fails the flow fails. Everything
after it catches its own exceptions and logs them. The reasoning, repeated in each task's
docstring, is that rows already in Redis are worth more than a failed flow. A missing correlation is a thinner payload; a missing history night is a gap; neither is an outage.
Note the order at the end. Our architecture diagram drew the S&P 500 and shares-float refreshes as branches off the start of the flow. Re-reading the code for this article, they actually run last, after the snapshot. That doesn't change any output, but it's a small example of why we re-trace diagrams against code rather than trusting them.
Choosing Tonight's Market Scan Symbols
The eligible universe is every enabled NYSE, NASDAQ and AMEX listing in
`inst_issues.symbol_reference`, excluding mutual funds and ETFs: about 6,744 symbols. Scanning all of them nightly would be too slow, so each night scans about 2,000, chosen at random from symbols outside a two-day cooldown.
The rotation is driven by three Redis keys:
scanner:used_symbols`, a sorted set scored by the time each symbol was last scanned. Anything scored within the last two days is ineligible.
`scanner:scheduled_symbols`, tonight's list. At the end of each run the service pre-generates the next night's 2,000 and stores them here.
`scanner:progress`, a hash holding the selected list, results so far and the current index,
with a 24-hour TTL.
There's a subtlety in the count. The flow's default `count` parameter is 1,000, but the
pre-generated list is built with 2,000, and when a scheduled list exists the service uses it
whole. So in practice every night scans about 2,000. If the cooldown ever exhausts the pool, the service falls back to the least recently scanned symbols, then to a random sample.
The progress hash makes the scan resumable. Results are flushed every 50 symbols, and if the flow is retried (or the machine restarts) the next run finds the hash and picks up at the saved index rather than starting over. The flow's `force_new_scan` parameter clears both the progress and the scheduled list when we want a clean start.
A consequence of the rotation: any one night covers less than a third of the universe. Seven
consecutive nights cover about 6,530 of the 6,744. That's why, until October 2026, AAPL or MSFT were often simply missing from the Pro Scanner. We come back to that below.
Concurrency is 12 symbols at a time for the US scan (four for an international variant, which
hits a rate-limited quote source first). An international run with `SCANNER_REGION=intl` uses its own key namespace and then merges its rows into the main payload so the same publish path picks them up.
Scanning One Symbol
For each symbol, `pricing/scanner_service` fetches 260 daily bars, applies the technical
indicators once over the whole frame, and then reads off what it needs.
Why 260 bars. The longest pattern lookback is 90 days, but one detector, the EMA-20 pullback, needs a 200-period EMA to confirm the trend. The scan used to fetch 120 bars, and the shared helper that adds detector columns refuses to emit `ema_200` on a frame shorter than 200 bars. That refusal is load-bearing: the indicator pipeline ends with a forward-fill, back-fill and fill-with-zero over every column, so an all-NaN EMA would become all zeros, making "close above the 200-day EMA" trivially true and reporting a confirmed uptrend on every short frame. With 120 bars the detector correctly found its column missing and returned nothing, for every symbol, every night. 260 leaves about 60 rows where the 200-day EMA is defined.
From the indicator frame the scan produces:
patterns: for each lookback of 5, 10, 30 and 90 days, the set of chart patterns active on
the latest bar (double top, triple bottom, head and shoulders, cup and handle and so on).
pattern_summary: the flat list of every active pattern.
pattern_signals: short-term daily setups from `detect_all_daily`, evaluated against SPY as a benchmark: breakouts, Bollinger squeeze expansions, pullbacks, each with a side, a confidence and setup-specific details such as relative volume.
ml_insights: a lightweight correlation check. For the benchmark price and for volume, it computes a 40-bar rolling correlation with the stock's close, then a z-score of the last 20
values against the whole series. Above 1.5 standard deviations the status is "Deviated
(Moderate)", above 2.0 "Deviated (Strong)", otherwise "Stable".
norm_prices: the last 20 closes, z-normalised. These are popped off each row and fed to a K-Means clustering (five clusters) across the whole scan, which becomes `discovered_patterns`: each cluster's centroid curve, its member symbols and a label from the centroid's slope and volatility ("Strong Upward Trend", "Tight Trading Range" and so on).
Only symbols with at least one pattern go into `patterns_found`. Every scanned symbol, pattern or not, goes into `all_symbols` with its name, which is what lets the history table record "scanned, nothing found" later.
The z-score baseline spans the whole fetched frame, which ties `ml_insights` to `SCAN_BARS`. Widening the frame would silently restate every z-score and status already in the payload. That's the main reason the heavier correlation work happens in a separate pass with its own frame.
What's in a Market Scan Row
Field | What it is | What we measured |
|---|---|---|
symbol, company_name | Identity; name from the Redis copy of company profiles | 100% |
market_cap | From the same profiles | 99% |
last_price | Latest Close price | 100%; spans $0.0005 to $1,752, so sub-dollar names need four decimals |
date | Date of the latest bar | Most rows match the scan date; a tail (48 of 934) is days to months behind |
patterns | Active patterns per lookback | Identical across all four lookbacks on 1,079 of 1,080 rows |
pattern_summary | Flat list of active patterns | Always present on `patterns_found` rows |
pattern_signals | Daily setups with side, confidence, details | Only 16–21% of rows populated |
ml_insights | Benchmark and volume correlation status | Two entries per row; 8% carry a "Deviated" status |
correlations | 1- to 5-year correlations (added by enrichment) | Filled for nearly every row after the retry pass |
predicted_patterns | Next-pattern predictions (added by enrichment) | 98% filled, but near-constant (see below) |
sector | Attached at publish time | Not in the raw scan; added from profiles |
The envelope around the rows carries `timestamp` (when the scan finished), `total_scanned`,
`all_symbols`, `discovered_patterns`, `timeframes` and, after enrichment, `enriched_at`.
Stage 3: Enrichment
The scan declares `correlations` and `predicted_patterns` on every row and populates neither. For a long time every row shipped `"correlations": {}` and `"predicted_patterns": []`. The scan's own comment explains why:
the full correlation analysis is too heavy to run inline for 2,000 symbols.
`pricing/scanner_enrichment` does that work afterwards, chained straight after the scan rather than given its own deployment, for two reasons: it needs the scan's output, and it reads the daily price cache the scan has just warmed (one-hour TTL). Run cold, it refetches every symbol through the slow path and takes an order of magnitude longer.
Correlations. For each symbol it builds a frame of the stock against a benchmark and a set of fundamentals (index price, EPS growth, net income, EPS, ROE, ROA, book value), and computes correlations over 1-, 3- and 5-year windows. A window is only emitted when the frame covers at least 90% of it, because a vendor's "five years" is about 1,254 bars against a nominal 1,260, and rejecting a window over six bars would drop data we genuinely have. The code knows about 10- and 20-year windows too, but the frame is capped at 1,250 bars, so those never fill.
The benchmark depends on the stock. Index membership comes first (S&P 500, then 400, then 600), then market cap from the scan row ($10 billion and up is large, $2 billion and up is mid). The small-cap index symbol returns zero bars from every provider on our plan, so the iShares small-cap ETF stands in, and the display name says so. Every row records `benchmark_source` (`index_membership`, `market_cap` or `default`), because the constituent lists aren't reliably available. On one run with Postgres unreachable, FMP answered the S&P 500 list with 503 symbols and the 400 and 600 lists with none, so every mid- and small-cap fell through to market cap. A reader can tell a constituent match from a size-bucket guess.
The pass is also where the scan's shortcuts would have been expensive. The correlation code it replaces decoded each whole fundamentals payload from Redis per symbol (key metrics alone is 100,000 rows) and fetched index constituents per symbol, about 3,000 calls for three fixed answers. Here each payload and each benchmark is loaded once.
Prices. Getting five years of daily bars for 1,000-odd symbols was the hardest part to make fast. The first version used the general-purpose price helper, which only serves its cache when the cached frame is at least as deep as requested. Seasoned symbols have 1,254 bars, but a young listing like ARM has 745, so it misses the cache forever and takes the cold path, which tries IBKR before FMP. IBKR serializes concurrent callers on one connection. Measured: 176 of 1,082 symbols in 63 minutes, with 127 timeouts at 120 seconds. That extrapolates to about six and a half hours for a task meant to finish the same evening.
The fix was to probe the cache at whatever depth it has (252 bars, one year, is enough), go
straight to FMP's end-of-day endpoint on a miss, and only fall back to the multi-provider path for listings FMP doesn't carry. The FMP calls go through the MCP client, which stalls once more than two requests are in flight [Article 3](ingest-by-domaim), so they sit behind a two-wide semaphore. One more trap surfaced here: the price helper defaults to a one-minute interval, which concatenated a day of minute bars onto the daily history. For ARM the last 252 rows of the frame were all stamped with the same date, so the "one year" window was one day of minute bars and its correlation came back 0.0. The pass now asks for daily bars explicitly.
Predicted patterns. Prediction is the entire cost of the pass, and it's CPU, not I/O. With
warm prices, fetching takes no time and the correlation maths about none; the predictor took 39 seconds per symbol with its original transition table and takes 6 to 11 seconds on 520 bars with the current one. All of it holds Python's GIL, so more async workers do nothing. The pass runs predictions in a process pool of up to 10 workers (cores minus two), with a few more async workers than processes so a symbol with prices already fetched is always waiting for a free core.
Two details matter for correctness, not speed. The frame passed to the predictor is trimmed to exactly 520 bars, because the price helper ignores its limit when serving a deeper cache entry and the predictor builds its table over whatever it's handed: on JPM, an untrimmed frame changed the published prediction and its support count went from 132 to 6. And the predictor was changed in October to be causal ("does this pattern appear within the next *h* bars", evaluated only on data available at the time) after we found the earlier version had lookahead.
Writing back. The pass re-reads `scanner:results:daily` before writing, because an
international merge may have rewritten it during the hour and a half the pass was running. It
assigns rather than appends each row's fields, so a re-run can't accumulate duplicates, and sets `enriched_at`.
The retry pass. A full pass takes about 103 minutes. In one measured run 23 symbols were missed. Only 5 were genuinely too short to yield a one-year window. The other 18 were a price timeout or a provider returning zero bars, and included GOOG, APH, GLW and WY, whose history obviously exists. So the flow runs a second pass over just the rows without correlations, which costs a couple of minutes.
Stage 4: Publishing the latest night to KV
`publish_scan_results_to_kv` exists for ordering, not as an optimization. A separate deployment, `daily-analytics-kv-sync`, also publishes this key, half an hour after the scan starts. Enrichment adds well over an hour and a half, so without a publish inside the scanner flow the app would see the previous night's rows, or un-enriched ones, every day. Publishing immediately after the fields are written makes the ordering a consequence of the code instead of two crons.
Before publishing it drops the local chat gateway's cached scanner responses (`cache:scanner_results:*`, one-hour TTL), because the publish reads through the gateway and would otherwise republish the pre-enrichment body it served earlier. The gateway attaches `sector` from company profiles, since raw scan rows have none. And the publish helper refuses to write a gateway response with `ok: false`. It used to wrap failures in a success envelope with no TTL, so an error body could sit in KV indefinitely on top of good data. A stale but valid payload beats a fresh error.
Three Writers, Last Write Wins
That ordering task is a symptom of a larger issue: three scheduled paths write the same KV key, `/api/analytics/scanner-results`.
Writer | When | Reads | Envelope |
|---|---|---|---|
Scanner Flow Publish | After enrichment, late evening | Gateway (LLM), which reads Redis | Gateway, which reads Redis | `{ok, data: {ok, result: {...}}}` |
`daily-analytics-kv-sync` | 21:30 daily | Same task, same gateway | Same |
`daily-market-data-sync` via `sync_scanner_prices.py` | 10:00 and 17:00 weekdays | Redis directly | `{ok, result: {...}}` |
Today these agree because they all read the same Redis key, and the in-flow publish is the one that lands after enrichment. But the setup is fragile in exactly the way the World P/E key was [Article 3](ingest-by-domain): the last writer decides what the app sees. Two details make
it worse than it looks:
The envelopes differ. Two writers double-wrap, one doesn't. Every consumer (the Worker, the history publisher, the app) has to normalize both, and one of our scripts has a comment that reads, accurately, "Two envelopes exist in the wild".
The architecture diagram listed different writers. It named an hourly Redis-to-KV sync and an accumulating five-day script. Re-tracing for this article, the hourly sync's scanner entry is commented out and the accumulating script isn't invoked by any flow. The diagram was right when drawn and drifted.
The Worker adds one more layer. If the scanner key is missing from KV entirely, the route falls
back to the shares-float key and reshapes those rows into scanner rows with no patterns. That keeps the grid from going blank, but it means "the scanner has rows and none of them have patterns" can mean "the scanner key expired", not "a quiet market". The Worker also unwraps one legacy envelope, promoting the scan's `timestamp` to a field called `scan_time`, which matters in a moment.
Stage 5: Snapshotting Predictions
Because each night scans a different third of the universe, the latest-night payload is a poor
answer to "what does the scanner say about AAPL?". Since 15 October 2026 the flow also keeps a week of results in D1, in a `scanner_history` table.
The history publisher reads the payload back **from the Worker**, not from Redis. That way each history row is exactly what the app rendered that night, including the `sector` attached during publish. Every scanned symbol gets a row under tonight's date: the full enriched row as JSON when it has patterns, just the name when it doesn't. Storing the misses was a deliberate choice: "we scanned it this week and found nothing" is different from "we haven't looked", and the app can show that difference.
The table's primary key is `(symbol, scan_date)`, with `has_patterns`, `market_cap`, `sector` and the payload. Writes are chunked at 12 rows per statement because eight columns times 12 rows is 96 bound parameters, under D1's limit of 100, and a request is capped at 600 rows; the publisher sends 500 at a time. Pruning keeps seven calendar days including tonight. The first night seeded 1,998 rows, 970 with patterns. There were no earlier nights anywhere to backfill from, since every previous store had been overwritten.
Why D1 and not KV
Everywhere else in this system we favour KV for wide, read-mostly payloads, and treat D1 writes as the budgeted resource ([Article 6](push-never-pull)). History goes the other way because of how it's read. The app needs "the newest row per symbol, older than tonight, ordered by market cap, 250 at a time", "this one symbol's newest row" and "every symbol scanned this week". In D1 each of those is a query over a `latest` common table expression. In KV each would mean fetching and filtering seven whole-night payloads on every request, or maintaining per-symbol keys and indexes by hand.
The write budget is small. About 2,000 rows a night is about 60,000 row writes a month, and with pruning about 120,000 D1 writes a month, against the 50 million included in the paid plan. Reads, which are where the queries cost, are about a thousand times cheaper per row than writes.
From Payload to Screen
The Pro Scanner (`MarketScannerNative`) loads the latest night from KV, then offers "Show earlier this week", which pages older nights from D1 250 rows at a time, each tagged with its scan date in amber. A search, or an assistant link to a symbol that isn't in tonight's rows, falls back to the single-symbol history lookup. Portfolio holdings link to the scanner using the week's symbol list.
Top Signals and the assistant's signal tools deliberately stay on the latest night. A week-old
breakout setup is stale in a way a week-old chart pattern isn't. Other consumers reach the scan indirectly. The Worker's sector-analysis route counts bullish and bearish pattern votes from the scanner payload to give each sector a technical direction.
The ML Insights sync (19:00) builds its deviation, predicted-pattern and correlation-activity panels from the scanner payload as a base, overlaid per symbol with anything the tech-analysis agent has written. It used to use the agent's outbox only and fall back to the scanner when the outbox was empty, but the outbox was rarely empty, just thin, so the panels showed 4 deviations instead of 93. That sync also sets a three-day TTL on `scanner:results:daily`. Each night's scan rewrites the key without a TTL, so this only bites if the scan fails three nights running, but then the key simply disappears.
Fields that Look Rich but are Not
When we built the iPad scanner's infographics, we measured every field against the live payload before giving it a visual encoding. Two of the richest-looking fields failed.
`patterns` can't encode timeframe. It's keyed by 5, 10, 30 and 90 days, which invites a
persistence heatmap. We built one first. It drew four identical cells on every card, because the pattern set was identical across all four lookbacks on 1,079 of 1,080 rows. The only information in it is how many patterns, not when.
`predicted_patterns` can't encode anything. It's 98% filled with three entries each, but the top prediction was one of just three names (double top, triple bottom, double bottom) on about 945 of 957 rows, at a median confidence of 0.133, with only 3% above 0.3. Drawing it would print the same low-confidence line on every card.
The fields that do vary carry the card. `ml_insights` draws a correlation "dumbbell" from historic to recent correlation on every card, and 553 of 934 move by more than 0.15. `last_price` and `market_cap` anchor it. The stale `date` tail gets a marker so an old close doesn't read as today's. `pattern_signals`, populated on only about one row in five, is useful precisely because it marks a minority out.
`pattern_signals` has artifacts of its own. A handful of symbols with split or listing problems
carry a previous high of a few cents against a close of $3 to $57, relative volume up to 4,777
times, and fire all four long setups at 0.99 confidence, so the producer ranks them first. The UI clamps relative volume at 20x and hides level distances beyond 50%; the producer itself isn't fixed yet. Confidence values cluster too (62 of 173 signals sat at exactly 0.85), so ranking needs tie-breaks.
We now run `aiim-mobile/scripts/verify-scanner-viz` before giving any scanner field an
encoding. It measures coverage and variance against the live payload and guards both rejected encodings. Sampling one row would have shipped the wrong chart twice.
The "As Of" Problem
The question users ask most about any panel is "how old is this?". For the scanner, the answer exists but has too many names:
Name | Where | Meaning |
|---|---|---|
timestamp | scan envelope | When the scan finished, local Eastern time, no offset |
enriched_at | scan envelope | When enrichment wrote back, as epoch seconds |
updated_at | KV envelope | When a sync published, in UTC |
scan_time | One Worker unwrap branch, and the accumulating script | The scan's `timestamp`, renamed |
date | each row | The date of that symbol's latest bar |
scan_date | History rows | The Eastern date of the night it was scanned |
These are four different facts: when the scan ran, when it was enriched, when it was published and how fresh each symbol's data is.
They're all legitimate. The problem is that readers pick one by name. The large-screen scanner reads `result.timestamp`. The phone scanner's type declares `scan_time`. A survey of every screen in October flagged the Predictions screen as reading `result.timestamp` where one Worker path supplies `scan_time`, so its "Nightly scan" footnote likely never renders. Which name a client sees depends on which envelope the last writer used.
The fix we've agreed is a shared `DataStamp` component with per-dataset staleness thresholds, showing "data as of" (the scan's own time, not the fetch time) on every screen. For the scanner, that means picking one field at the producer and having the Worker normalise every envelope to it, which is easier once there's one writer.
Debugging a stale scanner
When the scanner looks wrong, walk the lineage from the right-hand end: what the app got, then KV, then Redis, then the flow.
Which night is the app showing? Fetch `/api/analytics/scanner-results` and check the scan timestamp, `updated_at` and the envelope shape. A flat envelope means the market-data sync wrote last; double-wrapped means the scanner flow or the 21:30 sync did. No patterns on any row suggests the shares-float fallback, meaning the key is missing.
Did the latest run finish and enrich? Check `timestamp` and `enriched_at` in Redis. A timestamp from tonight with no `enriched_at`, or empty correlations, means enrichment failed and the flow carried on, as designed.
Is a scan stuck mid-way? A `scanner:progress` hash that exists outside the 21:00 window means a run died after selecting symbols. The next run will resume it. That's usually what you want, but `force_new_scan` clears it if not.
Did the flow run at all? If there's no Prefect run for `daily-market-scan`, look at the
serve process and at orphaned "running" rows that block overlap-guarded flows
Is history consistent? `/api/analytics/scanner-history/symbols` returns the earliest and
latest dates held. If `to` is behind the latest scan, the history publish failed; it logs and
carries on.
```bash
# Redis side: when did the scan finish and was it enriched?
redis-cli GET scanner:results:daily | jq '{timestamp, enriched_at, total_scanned, rows: (.patterns_found | length)}'
redis-cli EXISTS scanner:progress # 1 outside the scan window = a run died mid-scan
redis-cli TTL scanner:results:daily # a countdown here = TTL set by the ML insights sync since the last scan
```Most "stale scanner" reports so far have turned out to be one of three things: the symbol simply wasn't in tonight's third of the universe (now answered by history), the app's own cache, or a writer landing after the one we expected.
Takeaways
Put the ordering in the code. Publishing inside the scanner flow, after enrichment, removed a race between two crons that the app lost every day.
Non-fatal downstream stages are a feature. The scan is the product; correlations, history and snapshots are additions that shouldn't be able to take it down.
One key, one writer. Three writers with two envelopes work today only because they read the same source.
Measure fields before you draw them. `patterns` and `predicted_patterns` look rich in the schema and are near-constant in the data.
Use D1 where the reads are relational. Seven nights of history costs about 120,000 writes a month and turns "find AAPL this week" into one query.
Give "as of" one name. Four timestamps meaning four things is fine; six names for them isn't.
Re-trace your diagrams. Two of the writers on ours had quietly stopped writing.
