Architecture

Trace one response#

1. Schedule and fetch

   The feed catalog supplies per-feed intervals. The monotonic scheduler handles due feeds sequentially, with bounded retries and timeouts in the HTTP client.

2. Archive the bytes

   SHA-256 identifies byte content. New gzip files are written through a temporary file, fsync, and an atomic rename. Database failure still leads to an attempted filesystem archive.

3. Register the fetch

   A feed/hash pair reuses one snapshot; every fetch has its own observation. Duplicates skip parsing and preserve the snapshot’s earlier parse status.

4. Parse and commit

   A feed-specific parser returns domain records and warnings. Detailed records, compatibility projections, warnings, and parsed status are persisted transactionally. Failure rolls back that group; the handler marks the snapshot failed while its archived bytes remain.

5. Query and display

   PostgreSQL filters and aggregates before Polars/Streamlit render results. The dashboard never fetches CITA. Stopping it does not stop collection.

Inspect services and persistence#

docker compose ps
docker compose exec -T postgres pg_isready -U cita -d cita
docker compose exec -T collector python -m collector.healthcheck

Expected: PostgreSQL reports accepting connections. Collector health exits 0 for a recent successful or partial run, not complete feed coverage. The fixture helper does not create a collection-run row, so it alone cannot make this check pass.

Startup and query helpers initialize the schema. Although dashboard application queries read data, its current startup needs initialization privileges; a restricted read-only account is not a drop-in replacement.

Change the actual polling schedule#

In your own source checkout, edit FeedConfig.interval_seconds in config/feeds.py, then rebuild the collector.

docker compose build collector
docker compose up -d collector
docker compose logs --tail=30 collector

Expected: Startup logs enumerate feed intervals. COLLECTION_INTERVAL_SECONDS is not a global scheduler override. The active fetcher uses REQUEST_TIMEOUT = (10, 30); the general REQUEST_TIMEOUT_SECONDS variable does not control that path.

Run the source tests#

python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements/dev.txt
.venv/bin/python -m pytest -p no:cacheprovider

Expected: Pytest exits 0. The submitted snapshot has 55 parametrized cases across 13 files. Database-query tests use test doubles, not a live database. The quickstart fixture separately exercises real PostgreSQL.

Persistent alert functions/tests exist, but the main collector/dashboard flow does not call sync_alerts. Quality is computed at read time; historical unassessed fields do not preserve a versioned quality decision.