Schedule work without putting a secret in the browser
The seed script inserts 18 example feeds across English, French, and Arabic, then registers a pg_cron job every 15 minutes. pg_net makes an HTTP request to the pipeline service using the internal Compose hostname. At container start, Compose supplies the ingest token to Postgres as a setting; the request includes that value in the X-Ingest-Token header.
This choice keeps scheduling beside the database and avoids a separate cron service. It also ties the schedule to the initialized database volume: changing an init script does not automatically re-run it on an existing volume. For a different deployment, an external scheduler could call the same endpoint, but it would need network access and careful secret management.
SELECT cron.schedule('wassla-ingest', '*/15 * * * *',
$$SELECT net.http_post(
url := 'http://pipeline:8000/ingest/run',
body := jsonb_build_object('max_items', 50),
headers := jsonb_build_object(
'Content-Type', 'application/json',
'X-Ingest-Token', current_setting('app.ingest_token')
),
timeout_milliseconds := 300000
)$$
);Turn feed entries into a consistent record
For each active source, feeds.py fetches the configured feed URL with a timeout and a User-Agent. Redirects are followed one at a time, up to the configured limit, and only when the destination hostname belongs to one of the configured feed hosts. A response is rejected if its buffered body exceeds the configured byte limit. The limit is checked after httpx has received the body, so it is a validation threshold rather than a streaming download cap.
feedparser reads the XML. Wassla prefers an entry content field, then a summary or description; HTML tags are removed and whitespace is collapsed. Entries without both a link and title, or with fewer than 80 extracted body characters, are discarded. Kept bodies are truncated to the configured maximum, currently 12,000 characters. This avoids crawling publisher pages, at the cost of depending on how much text each RSS source exposes.
Deduplicate, order, and cap
Before fetching, the pipeline loads the set of URLs already in articles. It fetches all active feeds concurrently, removes entries whose URL is already known, and drops identical duplicates found within the fetched batches. Existing-article duplicates are counted; repeated entries within the same batch are simply discarded. The remaining items are ordered newest first, with undated entries last, and only then limited by max_items.
The URL is the identity key in both this check and the database uniqueness constraint. That is easy to explain and prevents ordinary repeats, but it does not canonicalize URLs: tracking parameters or alternate publisher URLs can represent the same story as different records. Fetching all feeds before applying max_items also means a small processing cap does not reduce the number of feed requests.
Failure is not yet a complete ingest report
The endpoint compares the supplied token with hmac.compare_digest and rejects missing or incorrect values. Individual article-processing errors are included in the response, but feed-fetch exceptions are collected by asyncio.gather and then skipped. As a result, a run can return successfully even when one or more feeds failed, without naming those feed failures in the response. The feed tests explicitly verify that failing feeds are skipped.
A useful next improvement would preserve a per-source outcome—success, timeout, invalid response, or parse failure—and expose it in logs or the response. In the current design, ingestion is bounded and mostly repeatable, but observability at the feed boundary is less complete than observability inside the model graph.