What it is
A containerized service that turns raw tabletop-RPG session transcripts into structured recaps. It runs as a Discord bot, stores transcripts and summaries in SQLite, and calls an LLM through a small client layer. The model is asked for one strict JSON object, so the pipeline can validate and repair output mechanically instead of parsing prose.
Design choices
- Strict contract. The summarizer receives the transcript, a speaker table, and the exact JSON keys to return. Anything outside the object is a failure, not a bonus.
- Grounding. The agent is instructed to read the relevant wiki pages and query the session database, and to mark facts as
(from wiki)or(from database). - Reproducible runtime. The service ships as a Docker image; configuration and data live on a mounted volume, so the image can be rebuilt without losing state.
- Evaluation. A harness captures a session window as a self-contained fixture, replays it through the summarizer, and stores the result next to a human-authored reference. The planned metrics are fact recall, hallucination rate, thread continuity, quote verbatim match, and grounding accuracy.
What I learned
LLM pipelines fail at the boundaries: a model that returns valid JSON can still miss a fact, and a prompt that works on one session can collapse on the next. The evaluation harness exists because “it looked fine” is not a quality measure. The scoring phase is the remaining work; the fixtures and replay path are already there.
Stack
Python, SQLite, Docker/Podman, Discord.py, httpx, and an OpenCode-based harness client.