Troubleshooting¶
The failures that have actually happened on this deployment.
A document is stuck at "processing"¶
The Celery worker restarted mid-task and the task was lost. The fix resets
anything over 30 minutes old to ready so it can be reprocessed.
Avoid it by checking before you restart:
Queued jobs never run, and nothing says why¶
Check that the worker is alive rather than crash-looping:
A crash-looping worker used to be invisible: restart: unless-stopped plus an
ImportError is a silent loop whose only symptom is that jobs do not run. It
went 69 restarts before anybody noticed. There is a healthcheck now
(inspect ping), so an unhealthy celery in docker compose ps is the
symptom to look for.
The commonest cause is a stale image: backend, celery and celery-beat
all build from ./backend, and rebuilding only backend leaves the worker on
old code. Use ./deploy.sh.
Scheduled jobs never run¶
Beat answers no ping — it is a clock, not a worker — so its healthcheck reads
/proc for a process with beat in its command line.
If beat is healthy and the worker is idle, suspect the Redis mix-up: the
backend and the worker sit on danvics_litellm, which has its own service
called redis, and Docker will resolve that one first. The queue then lives in
the LLM proxy's Redis while beat, on the default network only, publishes to
this one. REDIS_URL must name quiz-redis-1.
Every media read fails, and the bucket looks empty¶
InvalidAccessKeyId. Same story as Redis: danvics_milvus has a minio of
its own. This stack's MinIO carries the network alias quiz-minio, and
S3_ENDPOINT_URL must use it.
A backend worker fails on startup¶
One or two of the four uvicorn workers may log Application startup failed on
boot. The advisory lock around startup DDL was added to stop this, so it should
be rare now. The surviving workers serve all traffic.
Search returns noise¶
Almost always the embedding model. Every vector in the database was produced by
one model, and vectors from two different models are not comparable. This is
why PUT /admin/settings refuses to change embedding_model — see
Configuration.
If the model is right, check the reranker: POST /api/v1/admin/rerank/test. A
broken reranker cannot corrupt an order — it is a permutation and returns the
ids untouched on any failure — so a bad reranker makes results unimproved,
never wrong.
A 422 on an upload with no useful detail¶
Look in the backend log. Validation failures are logged with the shape of what arrived — content type, content length and the field names, never the content.
This was added after an iPhone upload failed twice with a 279-byte body that
said only "Field required" for body.file: the form had arrived without its
file part at all, most likely Safari sending a form minus an iCloud Drive
document that had never been downloaded. Invisible from the payload, obvious
from the headers.
A big upload gets a bare 413¶
nginx's client_max_body_size is below MAX_UPLOAD_SIZE. Raise it just above,
and the application returns a clear 400 instead.
Rate limits apply to the whole site at once¶
Every request looks like it came from the proxy. Two things must both be true:
nginx sets X-Forwarded-For overwriting whatever the client sent, and
uvicorn runs with --proxy-headers.
Somebody cannot sign in¶
See When sign-in breaks. Note that
POST /auth/login answering 410 is correct, not a fault — there is no
password door.
Logs¶
docker compose logs backend --tail=50
docker compose logs celery --tail=50
docker compose logs frontend --tail=50
Everything is also in Loki, queryable from Grafana on 127.0.0.1:3002.