Background work¶

Celery over Redis. One worker service, one beat.

celery   →  celery -A app.tasks worker --concurrency=2
beat     →  celery -A app.tasks beat

Both build from ./backend, which is why ./deploy.sh rebuilds all three.

Scale the worker, never the beat¶

The worker used to be worker --beat — one process doing both — on the reasoning that a single worker needs no lock. That holds exactly until there are two, and then every scheduled task fires twice: the embedding sweep twice, the topic claims twice, and purge_trashed_media twice, which is the one that removes files from an un-versioned bucket.

So: celery runs no scheduler and can be scaled freely. celery-beat is exactly one replica, always — in compose, and in whatever runs this next. On Kubernetes that means replicas: 1 with a Recreate strategy, not a rolling update: two beats overlapping for a few seconds during a deploy is the same double-fire, just rarer and harder to explain.

Beat queues work and performs none, so it needs no volumes and no concurrency.

The schedule¶

Task	Every	Does
`retry_missing_embeddings`	15 minutes	Sweeps up questions whose embedding failed at creation and would otherwise never be semantically searchable. Normally finds nothing.
`apply_topic_claims`	30 minutes	Applies topic claims that bypassed the two live sync points — bulk SQL, imports, restores. Also normally finds nothing.
`purge_trashed_media`	daily	The only scheduled job that destroys anything. Removes the stored bytes of images binned more than 30 days ago.

Timezone UTC.

purge_trashed_media is daily rather than hourly because nothing about it is urgent — the delay is the feature — and because a job that removes files should run seldom enough that a bad deploy is caught before its second run.

The tasks that are queued on demand¶

Task	Queued by
`process_pdf`	`POST /documents/upload`. Falls back to running inside the request if Celery or Redis is unreachable.
`extract_quiz`	`POST /drafts/extract`
`generate_article_draft`	`POST /articles/ai-draft` and `POST /articles/{id}/ai-refine`
`generate_article_cards`	`POST /articles/{id}/ai-cards`
`regenerate_embeddings`	Settings → embedding regeneration
`classify_question_difficulty`	Nothing. It exists and is registered but has no caller in the application.

Watching a job¶

Progress steps are pushed to a Redis list per job and polled by the front end from /jobs.

docker compose logs celery --tail=50
docker compose exec celery celery -A app.tasks inspect registered
docker compose exec celery celery -A app.tasks inspect active     # before any restart

Never restart while a task is running. A killed extraction leaves its document at processing for ever; see Troubleshooting.

The other singleton¶

Separately from Celery, the four uvicorn workers use a Redis SETNX lock (startup:singleton_lock, 300 s) so that exactly one of them starts the in-process scheduler and the embedding backfill thread at boot.