lenz-web/Agents.md

Agents

This codebase powers the born-digital edition of Johann Jacob Lenz’s correspondence. Every package acts like an agent in the editorial pipeline: ingesting the XML corpus, enriching it with scholarly context, and presenting the material on the public website. This document inventories those agents, how they cooperate, and where to extend them.

System lineage

• Entry point (lenz.go) – boots with config, clones or reuses the corpus Git repo via git/, parses TEI-like XML into memory (xmlmodels), wires the templating engine and cache, then starts the Fiber server.
• Data origin – XML, references, and tradition files live inside the checked-out repository (config.BaseDIR + config.GITPath). Changes arrive either through CLI restarts, filesystem watchers (dev), or GitHub webhooks (controllers/webhook.go).
• Request flow – Fiber receives HTTP traffic (server/), routes it through the controllers (controllers/*.go), pulls structured data from xmlmodels, renders Go templates (templating/, views/), and serves static assets through the custom filesystem middleware.

Agent catalog

1. Configuration & boot agent

• Scope – config/, lenz.go.
• Responsibilities
  • Merge config.dev.json, config.json, and KGPZ_* env vars, then enforce defaults for cache directories, bind address, webhook endpoint, etc.
  • Shape the runtime: debug mode flips Fiber logging on, turns on hot reloaders, and exposes the WebSocket live-reload port.
  • Stage the working directories (_cache/git, _cache/gnd, _cache/geo, _cache/search, data_bilder).
• Extension hooks
  • Add new configuration knobs to config.Config, plumb them through ConfigProvider, then surface them inside controllers via ctx.Locals("cfg", cfg).
  • Update Dockerfile or docker-compose.* when the runtime contract changes (ports, binaries, base path).

2. Corpus sync agent

• Scope – git/git.go, _cache/git, controllers/webhook.go.
• Responsibilities
  • Clone, validate, and update the corpus repository (default https://github.com/Theodor-Springmann-Stiftung/lenz-briefe) while keeping branch and hash bookkeeping (gitprovider.Commit).
  • Expose two lifecycle calls: OpenOrClone (during boot) and Pull (during webhook), each guarded by a global mutex to avoid concurrent git state.
  • On webhook delivery, verify X-Hub-Signature-256, pull the branch, trigger xmlmodels.Parse, and reset the cache store so visitors immediately see the update.
• Extension hooks
  • Add new repo validation rules by extending ValidateBranch or the webhook handler (e.g., topic filtering).
  • Emit observability metrics or notifications after Pull and parse success.

3. XML modeling agent

• Scope – xmlmodels/, xmlparsing/.
• Responsibilities
  • Parse data/xml/meta.xml, briefe.xml, references.xml, and traditions.xml concurrently into strongly typed structs (Meta, Letter, Tradition, PersonDef, LocationDef, etc.).
  • Provide a singleton Library (xmlmodels.Set/Get) that caches derived artefacts (sync.Map for year groupings, person/place lookups) and exposes helpers like NextPrev, Years, and Tradition.
  • Offload streaming tokenization, entity resolution, optional booleans, and TEI-ish constructs to xmlparsing (see parser.go, resolver.go, optionalbool.go, xmlsort.go).
• Extension hooks
  • Add new TEI registers by amending xmlmodels/library.go and xmlmodels/roots.go, then plugging in an xmlparsing.XMLParser for the new structure.
  • When introducing derived caches, use the built-in sync.Map to avoid reprocesing across requests.

4. Text rendering agent

• Scope – helpers/functions/textparse.go, helpers/functions/*.go.
• Responsibilities
  • Convert Letter.Content (inner XML) into semantic HTML fragments consumable by the templates: inline sidenotes, block notes, insertion markers, manuscript hands, printable line/pagination counters, etc.
  • Offer template-safe utilities (date math, slicing, string helpers, HTML escaping, embedder functions) injected via templating.Engine.AddFunc and Library.FuncMap().
• Extension hooks
  • Support new TEI tags or editorial conventions by updating the switch in Parse and by introducing helper Tokens operations when needed.
  • Register the helper with templates through templating.Engine.AddFunc("NewFunc", fn) to keep the template surface area explicit.

5. Presentation & asset agent

• Scope – templating/, views/, helpers/middleware.
• Responsibilities
  • Maintain the custom Go template engine (templating.Engine): layout registry, route registry, live-reload WebSocket (port 9000), and debug refresh/reset watchers (see watcher.go).
  • Deliver static artefacts (views/assets, fonts, paged.js bundle) through the fiber-aware filesystem middleware with cache-control + optional weak ETags.
  • Organize the view layer: views/layouts/default/root.gohtml for the shell, views/routes/**/head.gohtml & body.gohtml pairs for routes, views/routes/components for partials, and views/public as embeddable snippets/XSLT.
  • Ship the CSS/JS toolchain (Tailwind 4, DaisyUI, Vite) via views/package.json, vite.config.js, and transform/site.css, producing the compiled assets/ bundle that gets embedded at build time (views/embed*.go).
• Extension hooks
  • When adding routes, create a new directory under views/routes/<slug>/ with head and body templates and call c.Render("/<slug>/", data) from the matching controller.
  • For new global data, use Engine.Globals so layouts gain the values without per-controller plumbing.
  • In dev, use cfg.Debug=true so SetupRefreshWatcher reloads templates when files under views/routes or views/layouts change.

6. Delivery agent (HTTP server & controllers)

• Scope – server/, controllers/.
• Responsibilities
  • Configure Fiber with shared middlewares (compression, cache, logging/recover, custom cache key generator) and expose the Server struct that owns the templating.Engine and memory-backed cache store (github.com/gofiber/storage/memory).
  • Register web routes:
    • / redirects to /briefe.
    • /briefe lists letters grouped into historic ranges (see controllers/uebersicht.go).
    • /brief/:letter renders an individual letter with prev/next navigation, parsed body, and tradition apparatus (controllers/brief.go).
    • /datenschutz, /ausgabe/zitation, /ausgabe/edition, /kontakt render mostly static material via the template partials.
    • /assets/** serves CSS/JS/images with compression + ETag (middleware).
  • Expose Go structs (e.g., DateRange) to the templates so grouping logic stays in Go rather than HTML.
• Extension hooks
  • Add future APIs (e.g., JSON endpoints) inside controllers while reusing xmlmodels for data access.
  • Extend caching strategy by tweaking server/cache.go or CacheFunc (e.g., to drop caching for preview routes).

7. Update, cache & observability agent

• Scope – watcher.go, server/cache.go, controllers/webhook.go.
• Responsibilities
  • Implement hot-reload in debug builds: watchers monitor ./views/assets for refreshes and ./views/layouts, ./views/routes for full reloads, then call Engine.Refresh() or Engine.Reload() and reset the in-memory cache.
  • Provide manual cache reset helpers (ResetFunction, RefreshFunction) that controllers or tooling can reuse.
  • Support GitHub webhook-triggered cache invalidation (see agent 2).
• Extension hooks
  • Watch additional directories (e.g., content markdown) by appending to REFRESH_CHANGES / RESET_CHANGES.
  • Integrate metrics by instrumenting server.Server.Start/Stop and the webhook handler.

Data contracts

Type	File	Purpose
xmlmodels.Meta	data/xml/meta.xml	Chronological metadata per letter (sent/received actions, person/place refs, proofing flags). Supplies the timeline, filters, and prev/next logic.
xmlmodels.Letter	data/xml/briefe.xml	Carries the full textual content, pagination hints, and handwriting markers. Parsed into HTML via the text rendering agent.
xmlmodels.Tradition	data/xml/traditions.xml	Encodes the apparatus (Apps) for each letter so the UI can show textual variants and transmission notes.
xmlmodels.PersonDef, LocationDef, AppDef	data/xml/references.xml	Lookup tables for persons, places, and apparatus definitions, surfaced through the template func map (Person, Place, App).

All XML parsing goes through xmlparsing.XMLParser[T], so new record types only require defining a struct with xml tags plus a corresponding root type in xmlmodels/roots.go.

Request lifecycle (TL;DR)

1. Startup – config.Get() → gitprovider.OpenOrClone() → xmlmodels.Parse() populates the singleton library and caches.
2. Serve – server.New() builds Fiber + templating + cache; controllers.Register() binds routes.
3. Handle request – Controller retrieves data from xmlmodels.Get(), optionally transforms it (sorting, grouping), and renders a template. The templating engine merges controller data with globals and layout fragments, optionally injecting the live-reload script in debug mode.
4. Deliver – Responses are cached in memory (unless CacheFunc opts out) and static assets flow through the filesystem middleware with compression + weak validators.
5. Update corpus – GitHub webhook or manual restart pulls new XML content, re-runs xmlmodels.Parse, clears caches, and future requests see the updated edition.

Development workflow

• Go server – go run . (or air, etc.). Ensure _cache/git is writable; config defaults to _cache relative to repo root.
• Frontend assets – cd views && npm install && npm run dev during template work, or npm run build to regenerate views/assets. The Go build embeds the compiled assets unless built with -tags dev.
• Debugging – Set debug to true in config.dev.json. That turns on verbose logs, template reloaders, the WebSocket refresher, and disables static caching so CSS/JS changes land instantly.
• Deployment – Dockerfile builds a static binary (go build) and exposes port 8085. Mount _cache as a volume (or bake in) so git clones persist across container restarts.

Together these agents ensure that the scholarly corpus, textual enrichment logic, and presentation layer remain decoupled yet synchronized, making it straightforward to add new editorial features or publish new findings about Lenz’s letters.