43 cron jobs, one process: keeping a home-lab pipeline alive

Prefect runs the schedule, a homegrown governor runs everything else, and Redis lists hold

it together. Here's how that works, and the silent failures that shaped it.

AIIM's local pipeline has two runtimes, and they suit different kinds of work.

- Prefect fires 43 deployments on cron schedules. These are batch jobs with a clear start

and end: pull prices at 16:30, scan the market at 21:00, label the day's predictions at

22:30.

- The agent governor (`util/agent_governor.py`) keeps about ten long-running processes

alive. These are reactive: they block on a Redis queue, do work when a message arrives, and write a heartbeat so the governor knows they're still there.

The governor also supervises the Prefect process itself. That one entry in a Python dict

took a seven-hour outage to learn. This article covers how the two runtimes fit together,

what the schedule looks like across a week, and the failures (every one of them silent) that

turned a hobby-grade setup into something we trust to run unattended.

Why two pipeline runtimes at all

The obvious question is why we didn't put everything into Prefect, or everything into

agents.

Prefect is good at things that happen at a time: retries, run history, a UI showing what ran

and what failed, parameters, and the ability to kick off a run by hand. It's poor at things

that should *always be running*. A Prefect flow that loops forever is an anti-pattern: its run

never finishes, so the state table can't tell you anything useful, and every overlap guard

written against it gets confused.

Agents are the opposite. A process that `BLPOP`s a Redis list and reacts in milliseconds is

the right shape for the intraday stream, an on-demand graph search, or a worker that

pre-aggregates whatever the view mirror just refreshed. But agents have no natural place to

put "run this at 21:00 on weekdays", and an agent that does its own scheduling ends up

reimplementing cron badly.

So the split is: anything with a clock goes to Prefect, anything with a queue goes to an

agent, and the governor makes sure both kinds of process are alive.

Process topology

How the governor works

The governor is a single asyncio loop. Every 30 seconds it writes its own heartbeat to Redis,

then walks a static `AGENTS` dictionary and calls `check_agent` for each entry. Each entry

says how to start the process, where to log, which Redis key holds its heartbeat, and how

stale that heartbeat may get before the process counts as dead.

The thresholds differ a lot between agents, and each was chosen deliberately:

The intraday stream's two-minute threshold is the interesting one. A dead quote stream looks exactly like a quiet market to anything downstream: no new bars, no new signals. So it has to be caught in minutes, not half an hour. The agent keeps heartbeating even when its upstream feed is unreachable (it retries and publishes a status key instead), so a stale heartbeat from it means one thing only: the process died.

Beyond "is it alive", the governor has three more behaviours.

Interval tasks. Entries with an `interval` (the SEC RSS reader and cache maintenance) are

started when due, run to completion, and exit. If an instance is still running when the next

interval comes round, the governor skips that start rather than stacking a second copy.

Demand-based agents. The query agent and the Celery worker are marked `demand_based`. The governor checks `LLEN` on their inbox. If there's work and nothing running, it starts one; if the inbox has been empty for longer than `idle_timeout` (five minutes for the query agent), it shuts them down. Celery's ceiling comes from a `governor_limits.json` file that's re-read every sweep, so the worker count can be changed without restarting anything.

Stop grace. The default SIGTERM grace is five seconds. The stream agent gets twelve,

because on shutdown it flushes its time-of-day volume profiles, which took 3.0 seconds for 81 symbols and grows with the watchlist. Being SIGKILLed partway through that flush is exactly the data loss the flush exists to prevent.

The market-hours gate that moved

The stream agent was once the only agent with an `active_hours` window. The governor stoppedit at 16:05, because an ungated FMP poller burns one request a minute all night and all weekend.

That turned out to be wrong for a reason nobody had drawn on a diagram. The app's only

intraday endpoint is hosted inside that process. Stopping the agent also took away the

finished session at exactly the point it became worth reviewing, so every evening and all

weekend the intraday panel showed a connection error.

The window moved into the poll loop instead. Outside market hours the poller sleeps and

makes zero requests, while the HTTP API keeps serving bars from the retained session and

signals from Redis. The governor-level gate is still there for other agents that need it,

and it deliberately sits ahead of the staleness check. Otherwise the governor would read an intentionally stopped agent as a crash and respawn it within 120 seconds, fighting its own

gate.

The agent bus is just Redis lists

The agents don't talk to each other through Prefect. Each has up to three keys:

• `agent:{name}:inbox`: a list the agent `BLPOP`s from

• `agent:{name}:outbox`: a list it `LPUSH`es results to

• `agent:{name}:heartbeat`: a key holding a timestamp the governor reads

For example, the `pg_view_agent` watches a handful of Postgres views (company profiles,

ratios, analyst estimates, the sector-industry summary), copies them into Redis, and pushes a

message onto the aggregation agent's inbox. The aggregation agent picks it up, runs Cypher against Neo4j, and writes pre-aggregated results and bond risk rankings back to Redis, where the publish scripts find them.

The query agent's inbox has a sibling `cancel` key. A search the user has abandoned can be

called off rather than left to finish and write a result nobody will read.

Plain lists with no broker framework are boring, and boring is good here. You can inspect the

whole bus with `redis-cli`. A backlog is just `LLEN`. A dead agent leaves its inbox growing in

plain sight. There's no consumer-group state to reconcile after a crash, and no broker to

upgrade.

Lists have one weakness: `BLPOP` removes the message before the agent has processed it, so an agent that dies mid-message loses that message. We accept that because nearly all our messages are "something changed, recompute", which the next trigger repeats anyway. If a message ever carried something that can't be recomputed, `BLMOVE` into a processing list would be the fix.

Liveness is not progress

Heartbeats have one gap, and we've hit it. One afternoon the aggregation agent sat for hours with a fresh heartbeat and an undrained inbox. Its worker was stuck on HTTPS sockets in `CLOSE_WAIT` (the remote end had closed, the local end never noticed), while the heartbeat carried on from another task. The governor saw "alive". Issuers that should have been added to the bond risk results weren't, and it recovered on its own overnight.

A heartbeat proves the process is scheduled by the OS. It does not prove the process is

getting anything done. The cheap improvement is to heartbeat from the work loop itself, not

from a side task, so a stuck loop goes stale. The more thorough one is to alarm on inbox

depth times age: a non-empty inbox whose oldest message is older than the agent's normal

cycle.

Not every mystery gets solved. Two manual runs of the muni document scanner were SIGKILLed partway through a deep scan, at 40 and 59 minutes. It wasn't Prefect (no cancelling state), it wasn't the governor, and it wasn't memory: a later run peaked under 2 GB and finished. The cause is still unknown. Long statements are also slow: a 724-page bond statement takes about eleven minutes, and that's not a hang. That's the other half of "liveness isn't progress": a slow run and a stuck one look the same until you know how long the work should take.

A weekday, hour by hour

The day has a clear rhythm. Ingest clusters around the market close: index quotes at 16:20,

ETF quotes and the closing price update at 16:30. Compute follows once closing prices have

landed: intraday outcomes at 16:45, portfolio performance at 17:00, the rebalance check at

17:15. The heavy evening block (scan at 21:00, analytics publish at 21:30, prediction

labelling at 22:30) runs once the day's data is settled and the machine is otherwise idle.

Publish follows each compute stage, never the other way round.

Some spacing is deliberate. The 17:15 rebalance runs after the 17:00 performance roll-forward because it reads the NAV that job writes. The 22:30 labelling runs after the 22:00 analyst estimates and model memory sync, so it doesn't compete with them for the database. Prefect has no cross-deployment dependencies in this setup, so the gaps between start times are the dependency graph. That works, but it's brittle: if the 21:00 scan runs long, the 21:30 publish ships the previous night's scan. A future improvement is to have the scan trigger its own publish, which the scanner flow already does for its main payload.

The rest of the week

Not everything is daily. The weekly and monthly jobs mostly fill the weekend, when the

market is closed and the upstream APIs are quiet:

Market pulse also keeps running at the weekend, every four hours at :15 rather than every 30

minutes, so the app's headline tiles never look abandoned on a Sunday.

The weekend layout has a side effect that bit us once. The economics sync runs hourly from

08:00 to 18:00, **Monday to Friday only**. Its KV keys were written with a 24-hour TTL. From

Friday 18:00 to Monday 08:00 is 62 hours, so every weekend the economics screens went blank. The rule we now apply everywhere: a TTL must outlast the longest gap in the producer's schedule, not its typical interval.

The same investigation found two more traps in that flow. First, two different scripts in

the repo share the module name `sync_economics_data` and write the same KV keys in

incompatible shapes. Which one `import` picked depended on the order of entries in

`sys.path`, so reordering a list literal would have swapped producers behind a green flow

run. The flow now loads the right file by explicit path. Second, the flow captures `print`

output into Prefect's run log but not stdlib `logging`. A script that reports through

`logging` can fail completely and still leave a clean, COMPLETED run log behind.

Failure 1: the scheduler died and nobody noticed

All 43 schedules live in one `prefect aserve` process. None of the deployments is attached to

a work pool, which means the two `prefect worker` processes running alongside can't pick

them up. The workers being up tells you nothing about whether schedules are firing.

At first `deployments` was started once, with `nohup`, and nothing supervised it. One

night the Prefect server's metadata database became unreachable for a while. The server

returned 500s, the runner's poll for scheduled runs raised an exception, and the serve

process exited. On the way out it got another 500 inside its shutdown path, which pauses

every schedule. It could just as easily have left deployments paused with no process alive

to unpause them; we checked, and that time it hadn't.

The network recovered on its own and the Prefect server was healthy within hours. The serve

process stayed dead for about seven hours. No alert fired. The only evidence was a log

file that had stopped growing.

The fix was to make the serve process a governor-managed entry, with three settings that each matter:

The restart throttle sits after the market-hours gate in the code, on purpose: it must

never delay a stop, only a start.

We also set `limit=6` on `aserve`. The default is unlimited, and after a four-day outage the

restart submitted 84 late runs at once. Six is deliberately below the core count, because

many flows fan out internally.

The real trigger was database timeouts

The recurring cause wasn't a Prefect bug. The server's log showed `asyncpg` connection

timeouts against its metadata database, which sits on another machine on the LAN. One

measurement captured 57 `TimeoutError`s in a 130-second window, followed by full recovery

(a direct connect in 0.17 s). Prefect's defaults are a 5-second connection timeout and a

10-second query timeout, which is tight for that link. Raising them to 30 and 60 seconds in

the startup script, and in the profile, was the cheap fix, and it has held.

At the same time we pinned Prefect. The requirements file had been unpinned, and the

environment had drifted to an older minor version than we thought. Upgrading runs an Alembic migration against the shared metadata database, so the rule now is: take a `pg_dump` before any Prefect version bump.

Failure 2: zombie runs that disable a flow forever

Prefect leaves a flow run in `RUNNING` forever if its process dies without reporting a

terminal state: a hard kill, the machine sleeping, an OOM. Several of our flows guard against

overlap with "skip if another run of this flow is RUNNING". Put those together and a single

orphaned row disables the flow permanently, with no error anywhere.

When we went looking, we found 55 orphaned RUNNING rows going back three months. Five were sitting in front of the nightly app-sync flow, which had therefore published nothing for over a week. Every skipped run reported success, so the Prefect UI was a column of green ticks. The only visible symptom was two blank screens in the app, and they appeared only once the KV keys aged past their 3-day TTL. The TTL had hidden the outage for three days, then revealed it far from its cause.

The fix is a helper, `other_live_runs(flow_name, stale_after)` in `orchestration/flow_concurrency`, which cancels runs older than a per-flow threshold

instead of deferring to them: 15 minutes for pricing, 45 for the economics sync and model

performance. The pricing flow already had a version of this idea, with a classic bug: it

removed items from the list it was iterating over, so it only ever cleared every other stuck

run.

Two other bugs turned up the same day, both from the same "nothing errored" family:

• A gateway helper cached a `None` Redis client after one failed connection, and its re-init

  check tested whether the key was present, not whether the value was usable. One Redis blip wedged every Redis-backed gateway endpoint until a restart. It now retries on a 10-second cooldown.

• The analytics KV sync logged `ok: false` responses from the gateway, then published them anyway, inside its own `ok: true` envelope, with a TTL of zero (never expire). That

  poisoned the sector analysis key on top of good data until someone noticed. It now

  refuses to publish a failed payload.

A high RUNNING count is mostly stale rows, not load. Right after one clean restart there were 67 RUNNING rows, 29 of them more than a week old, but only seven live `prefect.engine` processes. Likewise, 34 of 37 PENDING rows were week-old submissions that had never left "Submitting". The state table and the process table disagreed by a factor of six.

Age-bucket the runs and count processes before concluding the box is overloaded.

Failure 3: the governor counts but doesn't reappear

The governor's liveness check is `len(running_instances) > 0`. One live serve process

satisfies that, so a second is never noticed. We once found two `deployments.py` processes,

both re-parented to PID 1, started seven seconds apart. Both registered the same deployments and both polled for due runs. It showed up as duplicate runs of the same deployment seconds apart: one market scan went RUNNING, then CRASHED a minute later, then RUNNING again four seconds after that.

During that episode, Prefect's `/api/health` answered in 2 ms while the deployments filter

endpoint returned 500s or hung for minutes. Health checks are not evidence that a scheduler is working. The fix we want, and haven't shipped, is for the governor to treat "more than one" as a fault for singleton entries and kill the newer copies.

The timezone trap

Every deployment uses `CronSchedule(..., timezone="America/New_York")` except the two

four-hourly muni jobs, which pass a bare `cron=`. Prefect stores that with no timezone and

evaluates it in UTC. For an every-four-hours job the difference doesn't matter. But

copying one of those two as a template for a new daily job would land it four or five hours

off, depending on daylight saving. So the exception is documented in a comment next to them and in the schedule table, rather than "fixed" by a change that would move their run times.

All the other schedules were converted to explicit Eastern time in October 2026. Before

that, several ran on bare UTC crons, so their Eastern start times moved by an hour whenever

daylight saving changed, while the market close they depend on didn't. The same review

found and deleted a stale duplicate of the economics sync deployment.

Alternatives considered

Work pools and workers. This is the textbook Prefect setup: deployments in a pool,

workers polling it. It would let several machines share the load and survive one dying. We

haven't moved because the flows share a lot of local state (Redis on localhost, file caches,

the venv), so a second machine wouldn't really be interchangeable. A pool would add a moving part without removing the single point of failure.

launchd or systemd for the serve process. A real OS supervisor would be more robust than a Python loop. But the governor already exists, already handles restart throttling, and

already logs in the same place as everything else. Splitting supervision across two systems

would mean two places to look.

Moving schedules to the cloud. We've started shadow-running two publish flows (market

pulse and the portfolio universe closes) on Cloudflare Workflows alongside their Prefect

versions, comparing outputs before any cut-over. Anything that only needs FMP and the edge stores can move; anything that needs Neo4j, the local Postgres or IBKR can't.

The runbook

1. pgrep -f orchestration/deployments Is the serve process alive, and is there

exactly one? Don't trust worker status or `/api/health`.

2. RUNNING rows for that flow, bucketed by age. An orphan older than the flow's normal

runtime means the overlap guard is holding it.

3. Count `prefect.engine` processes before blaming load.

4. The log of the serve process. Rename it before restarting, because the startup script

truncates it with `>`, which destroys the crash evidence.

5. Only then, the schedule itself: timezone, cron expression, whether it's paused.

When an app screen is empty, work from the other end: check the KV key's `updated_at`, then the RUNNING rows for the flow that produces it, then whether a *second* producer writes the same key. That last one is the subject of [Article 6]

Takeways

• Put clocked work in a scheduler and queued work in agents, and supervise both from one place.

• If a scheduler's deployments have no work pool, the workers aren't your safety net.

• Supervise the process that owns the schedules.

• Overlap guards need an expiry. "Skip if RUNNING" without a staleness cutoff turns one crash into a permanent outage that reports success.

• Throttle restarts and cap concurrency on recovery. The backlog after an outage can do more damage than the outage did.

• Heartbeats measure liveness, not progress. Heartbeat from the work loop, and watch inbox age.

• A TTL must outlast the longest gap in its producer's schedule, weekends included.

• Health endpoints answer quickly while the work hangs. Probe the thing you actually depend on.