Trace the path through the graph

At startup, main.py calls setup_tracing before importing the graph and model modules. The OpenInference LangChain instrumentor exports spans over OTLP to the local Phoenix service. Graph nodes and LLM calls add tags such as ingest, article, eval, and task:summarize, so a developer can follow a request from the HTTP route into its model operations.

Tracing supports debugging and latency analysis, but it can also collect prompts, article text, and model completions. The supplied Compose configuration disables Phoenix authentication and binds its published ports to localhost, which is convenient for local work, not a production access policy. Treat traces as potentially sensitive data and control retention and access accordingly.

Experiments measure selected behaviors

make evals runs the article graph without persistence against a small JSONL dataset and records a Phoenix experiment. Structural checks verify that configured languages exist, summaries stay under 150 words, and evaluation scores and verdicts have the expected shape. With a real provider, the script also checks exact category labels, looks for Arabic script, and runs an LLM judge using the pipeline evaluation rubric.

These checks are useful regression probes, not a complete measure of translation quality, factual accuracy, or retrieval usefulness. The dataset contains only a handful of examples, and the judge is another call using the configured evaluator task. The script records experiments for comparison; it does not impose a score threshold that automatically blocks a release.

Tests verify boundaries with fakes

Pipeline tests use a fake asyncpg pool and deterministic chat and embedding providers to exercise graph states, SQL calls, route validation, and error handling without a live database or paid model API. Web tests mock Nuxt fetches and router behavior, then check page and component contracts. This makes many edge cases fast and repeatable.

A passing unit suite therefore proves behavior at the tested boundaries, not that a live provider returns valid language, that pgvector ranking is useful on real news, or that every Compose service recovers from failure. Coverage percentages describe executed code paths; they do not measure editorial quality. A full-system evaluation still needs representative data and human review.

Reproduce, inspect, and extend

For a local smoke run, use the stub providers and make up; for model behavior, supply provider credentials and ingest a small capped batch. Inspect the resulting spans in Phoenix, compare search hits with their source pages, and append carefully chosen cases to the evaluation dataset. Before changing prompts or models, keep a baseline experiment so regressions are visible rather than remembered.

The design is intentionally understandable end to end: RSS is the input contract, the article graph makes transformations explicit, Postgres stores text and vectors together, the read-only web proxy separates readers from ingestion, and Phoenix plus tests make selected behavior inspectable. Its limits are equally important: feed failures are not fully surfaced, review scores do not verify whole translated bodies, citations are not validated, embedding model versions are not stored, and the evaluation set is small.

The complete code, setup instructions, diagrams, and tests are in the Wassla repository. The system is a reproducible engineering example, not a claim that generated news summaries are authoritative. Questions, corrections, and alternative design choices are welcome.