PedsHub Deployment and Operations Guide¶

Docker Compose Setup¶

The application runs as 5 services defined in docker-compose.yml:

Service	Image	Purpose	Port
`postgres`	`pgvector/pgvector:pg16`	PostgreSQL with pgvector extension	Internal only
`redis`	`redis:7-alpine`	Caching, quiz progress, job tracking, Celery broker	Internal only
`backend`	Built from `./backend`	FastAPI app (uvicorn, 4 workers)	Internal only
`celery`	Built from `./backend` (same image)	Celery worker (concurrency=2)	None
`frontend`	Built from `./frontend`	Nginx serving React SPA	`127.0.0.1:8081:80`

Volumes¶

Volume	Used by	Purpose
`postgres_data`	postgres	Database persistence
`redis_data`	redis	Redis persistence
`uploads_data`	backend, celery	Uploaded PDF files and extracted images
`chroma_data`	backend, celery	ChromaDB vector store

Health Checks¶

postgres: pg_isready -U pedquiz every 5s, 10 retries.
Backend and celery depend on postgres being healthy (condition: service_healthy) and redis being started.

Startup Order¶

Redis and Postgres start first.
Backend and Celery wait for Postgres to be healthy.
Frontend waits for backend to start.

Environment Variables¶

Backend (`backend/.env`)¶

Variable	Default	Description
`DATABASE_URL`	`sqlite:///./quiz.db`	PostgreSQL connection string. Production: `postgresql://user:pass@postgres:5432/pedquiz`
`SECRET_KEY`	`change-me-...`	JWT signing key. Generate with `openssl rand -hex 32`
`ALGORITHM`	`HS256`	JWT algorithm
`ACCESS_TOKEN_EXPIRE_MINUTES`	`1440`	Token lifetime (default 24 hours)
`REDIS_URL`	`redis://localhost:6379/0`	Redis connection URL
`LITELLM_MODEL`	`gpt-4o-mini`	Default LLM model for extraction
`LITELLM_API_KEY`	(empty)	API key for LiteLLM proxy or provider
`LITELLM_API_BASE`	(empty)	Custom API base URL (for LiteLLM proxy)
`LITELLM_EMBEDDING_MODEL`	(empty)	Model for semantic search embeddings
`OPENAI_API_KEY`	(empty)	OpenAI API key (if using OpenAI models directly)
`ELEVENLABS_API_KEY`	(empty)	ElevenLabs API key for TTS voices
`GOOGLE_TTS_API_KEY`	(empty)	Google Cloud TTS API key
`AWS_ACCESS_KEY_ID`	(empty)	AWS credentials for Bedrock / Polly
`AWS_SECRET_ACCESS_KEY`	(empty)	AWS secret key
`AWS_REGION`	`us-east-1`	AWS region
`AWS_BEDROCK_REGION`	`us-east-1`	AWS Bedrock region
`EMBEDDING_DIMENSIONS`	`1024`	Vector embedding dimension size
`CHROMA_PERSIST_DIR`	`/app/chroma_data`	ChromaDB storage directory
`MAIL_USERNAME`	(empty)	SMTP username
`MAIL_PASSWORD`	(empty)	SMTP password
`MAIL_FROM`	(empty)	From address for emails
`MAIL_PORT`	`587`	SMTP port
`MAIL_SERVER`	`smtp.gmail.com`	SMTP server hostname
`MAIL_STARTTLS`	`true`	Use STARTTLS
`MAIL_SSL_TLS`	`false`	Use SSL/TLS
`UPLOAD_DIR`	`/app/uploads`	Upload storage directory
`MAX_UPLOAD_SIZE`	`524288000`	Max upload size in bytes (500MB)
`APP_URL`	`https://quiz.danvics.com`	Public URL (used in emails, links)
`CAP_SECRET_KEY`	(empty)	Cap secret key. Leave blank to disable captcha
`ADMIN_EMAIL`	(empty)	Email address for contact form submissions

Frontend (`frontend/.env`)¶

Variable	Description
`CAP_SITE_KEY`	Cap site key (public). Injected at runtime via `docker-entrypoint.sh` into `window.__APP_CONFIG__`

HTTPS Setup with Caddy¶

The frontend binds to 127.0.0.1:8081 (not exposed externally). Use a reverse proxy for HTTPS.

Example Caddyfile:

quiz.example.com {
    reverse_proxy localhost:8081
}

Caddy handles automatic HTTPS certificate provisioning via Let's Encrypt. No additional TLS configuration is needed.

For setups where the backend needs to be accessed directly (rare):

quiz.example.com {
    reverse_proxy localhost:8081

    # Optional: direct backend access for debugging
    handle_path /api-direct/* {
        reverse_proxy localhost:8000
    }
}

Note: The frontend's Nginx already proxies /api to the backend container internally, so only the frontend port needs to be exposed.

Rebuilding¶

When to Rebuild vs Restart¶

Change	Action
Backend Python code	Rebuild backend: `docker compose build backend`
Frontend source code	Rebuild frontend: `docker compose build frontend`
Backend `.env` changes	Restart only: `docker compose restart backend celery`
Frontend `.env` changes	Restart only: `docker compose restart frontend`
`requirements.txt` changes	Rebuild backend: `docker compose build backend`
`package.json` changes	Rebuild frontend: `docker compose build frontend`
`docker-compose.yml` changes	`docker compose up -d` (recreates changed services)

Celery shares the backend image¶

The celery service builds from ./backend — the same Dockerfile as backend. Rebuilding backend requires rebuilding celery as well:

docker compose build backend
docker compose up -d backend celery

Or build both explicitly:

docker compose build backend celery
docker compose up -d

Stale code / cache issues¶

If code changes are not reflected after a rebuild, use --no-cache:

docker compose build --no-cache backend
docker compose build --no-cache frontend

This forces Docker to re-run all build steps, including pip install and npm ci.

Database¶

PostgreSQL with pgvector¶

The database uses pgvector/pgvector:pg16 which includes the vector extension. On startup, the backend runs:

CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE questions ADD COLUMN IF NOT EXISTS embedding vector(1024);
CREATE INDEX IF NOT EXISTS questions_embedding_hnsw
    ON questions USING hnsw (embedding vector_cosine_ops);

Connection Pool¶

SQLAlchemy engine is configured with:

engine = create_engine(settings.DATABASE_URL, pool_pre_ping=True, pool_recycle=300)

pool_pre_ping=True — Tests connections before use. Prevents "connection closed" errors after database restarts.
pool_recycle=300 — Recycles connections every 5 minutes. Prevents stale connections in long-running processes (Celery workers).

Backups¶

# Dump the database
docker compose exec postgres pg_dump -U pedquiz pedquiz > backup_$(date +%Y%m%d).sql

# Restore from backup
docker compose exec -T postgres psql -U pedquiz pedquiz < backup_20240101.sql

For automated backups, add a cron job on the host:

0 2 * * * cd /home/danvics/docker/quiz && docker compose exec -T postgres pg_dump -U pedquiz pedquiz | gzip > /backups/pedquiz_$(date +\%Y\%m\%d).sql.gz

Monitoring¶

Checking Logs¶

# Backend logs (FastAPI + uvicorn)
docker compose logs -f backend

# Celery worker logs (extraction jobs, embeddings)
docker compose logs -f celery

# All services
docker compose logs -f

# Last 100 lines
docker compose logs --tail=100 backend

Common Errors and Fixes¶

Error	Cause	Fix
`connection already closed`	Stale DB connection in Celery	Already handled by `pool_pre_ping` and `pool_recycle`. If persistent, restart celery
`lock timeout` on startup	Previous killed process holding DDL locks	Backend auto-terminates stale connections and retries 3 times with 5s delay
`429 / rate limit` from LLM API	Too many extraction requests	Reduce celery concurrency or add rate limiting config
`embedding failed`	Embedding API rate limit or model issue	Check LITELLM_EMBEDDING_MODEL config, verify API key
`CORS error` in browser	Frontend not going through Nginx proxy	Ensure requests go through the frontend's Nginx (which proxies /api)
`502 Bad Gateway`	Backend not ready yet	Wait for backend to start, check `docker compose logs backend`

Troubleshooting¶

Lock Timeout on Startup¶

The backend runs DDL migrations (ALTER TABLE, CREATE INDEX) at startup. If a previous instance was killed while holding a lock, the new instance will hang. The startup code handles this automatically:

Terminates stale idle in transaction connections older than 30 seconds.
Sets lock_timeout = '15s' before running DDL.
Retries up to 3 times with 5-second delays if a lock timeout occurs.

If this still fails, manually kill stale connections:

docker compose exec postgres psql -U pedquiz -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction' AND pid <> pg_backend_pid();"

Singleton Lock for Startup Tasks¶

The backend uses a Redis-based singleton lock (startup:singleton_lock, 5-minute TTL) to ensure that only one of the 4 uvicorn workers runs the scheduler and backfill tasks. If Redis is unavailable, it assumes single-worker mode and runs everything.

Stale DB Connections in Celery Tasks¶

Celery workers are long-running processes. Without pool_recycle=300, connections can go stale (especially after postgres restarts). The pool_pre_ping=True setting validates connections before use. If database errors appear in celery logs after a postgres restart, restart the celery service:

docker compose restart celery

Embedding Batch Size / Rate Limits¶

Embedding generation happens during quiz extraction and on startup (backfill). If the embedding API has rate limits, failures may appear in the celery logs. The embedding service handles individual failures gracefully (logs and continues). To re-embed all questions:

docker compose exec backend python manage.py reembed

Docker Build Cache Issues¶

If Python dependencies or JS packages have been updated but the build is reusing cached layers:

# Force fresh install of all dependencies
docker compose build --no-cache backend frontend
docker compose up -d

Scaling¶

Worker Counts¶

Backend (uvicorn): Runs with --workers 4. For higher throughput, increase in docker-compose.yml. Each worker runs the startup lifespan (idempotent DDL + singleton lock for scheduler).
Celery: Runs with --concurrency=2. This controls how many extraction jobs run in parallel. Increase for faster throughput, but watch LLM API rate limits.

Redis Memory¶

Redis stores: - Quiz progress (per-user, per-quiz) - Extraction job status and steps - Celery broker messages - Singleton lock - User job lists

For most deployments, default Redis memory is sufficient. Monitor with:

docker compose exec redis redis-cli info memory

ChromaDB Storage¶

ChromaDB stores question embeddings for semantic search. Storage grows with the number of questions. The chroma_data volume persists this data. Monitor disk usage:

docker system df -v | grep chroma

Updating¶

Standard Update Flow¶

cd /home/danvics/docker/quiz
git pull
docker compose build backend frontend
docker compose up -d

Migration Safety¶

Database migrations run automatically on startup via Base.metadata.create_all() and explicit DDL in setup_pgvector(). These are idempotent: - CREATE TABLE IF NOT EXISTS - ADD COLUMN IF NOT EXISTS - CREATE INDEX IF NOT EXISTS

DDL statements use SET lock_timeout = '15s' and retry up to 3 times to handle concurrent lock contention from Celery workers or other uvicorn processes.

Zero-Downtime Updates¶

For zero-downtime updates (if needed):

# Build new images first
docker compose build backend frontend

# Restart one service at a time
docker compose up -d --no-deps backend
docker compose up -d --no-deps celery
docker compose up -d --no-deps frontend

CLI Tools¶

The manage.py script provides management commands. Run inside the backend container:

docker compose exec backend python manage.py <command>

Commands¶

`check-links`¶

Lists every pointer in the library that aims at a retired or missing article or section: [[id|…]] markers in prose, question key points, study-plan blocks, flashcards, collections, exam links and question reading links. Exits non-zero when anything is listed, so it can gate a deploy. References are expected to point at the article that answers them; redirect rows exist only so old URLs still land.

docker compose exec backend python -m app.cli check-links

`list-users`¶

List all users with ID, email, role, verified status, and name.

docker compose exec backend python manage.py list-users

Output:

  ID  Email                                     Role          Verified  Name
------------------------------------------------------------------------------------------
   1  admin@example.com                         admin         yes       Admin
   2  user@example.com                          user          yes       John

`reembed`¶

Regenerate all question embeddings. Useful after changing the embedding model. Clears existing embeddings and re-generates them one by one.

docker compose exec backend python manage.py reembed

`extract <section_id> [options]`¶

Run quiz extraction from a section. Can run inline (blocking) or in background (via Celery).

# Inline extraction (blocking, shows live output)
docker compose exec backend python manage.py extract 5

# With options
docker compose exec backend python manage.py extract 5 --title "Chapter 3 Quiz" --mode learning --user admin@example.com

# Background (via Celery)
docker compose exec backend python manage.py extract 5 --bg

Options: - --title — Quiz title (default: auto-generated from section name) - --mode — timed or learning (default: timed) - --user — User email to assign the quiz to (default: first admin user) - --bg — Run in background via Celery

`list-sections [doc_id]`¶

List all documents and their sections. Optionally filter by document ID.

# All documents
docker compose exec backend python manage.py list-sections

# Specific document
docker compose exec backend python manage.py list-sections 3

Output:

Doc 3: PREP_2024.pdf (ready, 450 pages)
  Section 5: 'Chapter 1'  pages 1-50
  Section 6: 'Chapter 2'  pages 51-120

`jobs [--user email]`¶

Show recent extraction jobs from Redis. Optionally filter by user email.

# All users' jobs
docker compose exec backend python manage.py jobs

# Specific user
docker compose exec backend python manage.py jobs --user admin@example.com

Output:

admin@example.com:
  [completed ] Quiz: Chapter 1                           steps= 12 quiz_id=7
              Saved 25 questions to quiz
  [running   ] Quiz: Chapter 2                           steps=  5
              Extracting questions from pages 51-80...

PedsHub Deployment and Operations Guide¶

Docker Compose Setup¶

Volumes¶

Health Checks¶

Startup Order¶

Environment Variables¶

Backend (backend/.env)¶

Frontend (frontend/.env)¶

HTTPS Setup with Caddy¶

Rebuilding¶

When to Rebuild vs Restart¶

Celery shares the backend image¶

Stale code / cache issues¶

Database¶

PostgreSQL with pgvector¶

Connection Pool¶

Backups¶

Monitoring¶

Checking Logs¶

Common Errors and Fixes¶

Troubleshooting¶

Lock Timeout on Startup¶

Singleton Lock for Startup Tasks¶

Stale DB Connections in Celery Tasks¶

Embedding Batch Size / Rate Limits¶

Docker Build Cache Issues¶

Scaling¶

Worker Counts¶

Redis Memory¶

ChromaDB Storage¶

Updating¶

Standard Update Flow¶

Migration Safety¶

Zero-Downtime Updates¶

CLI Tools¶

Commands¶

check-links¶

list-users¶

reembed¶

extract <section_id> [options]¶

list-sections [doc_id]¶

jobs [--user email]¶

Backend (`backend/.env`)¶

Frontend (`frontend/.env`)¶

`check-links`¶

`list-users`¶

`reembed`¶

`extract <section_id> [options]`¶

`list-sections [doc_id]`¶

`jobs [--user email]`¶