[Unreleased]

2026-07-08 — per-user login with roles (Zitadel), guest sees Insights only

Replaced the single shared password with per-user accounts via Zitadel (login.dream-machine.de — the same login as the KulturRadar web app). Two roles: admins see everything; Insights guests see only the Insights (analytics) pages — the scraper/curator/settings/explorer surfaces are hidden and their APIs return 403. Enforced both in the UI (nav + page gating) and server-side on every API route.
User administration (create accounts, reset passwords, grant the admin/insights role) is done in the Zitadel console — no in-app user management to maintain.
The machine-to-machine password path (k8s CronJobs, scripts) is unchanged, so the daily scrape/curator jobs keep running untouched.

2026-07-03 — curator tag precision: "familie" is kids/family-only

familie/kinder tags no longer leak onto adult events. The tag prompt listed familie as valid vocabulary but never defined it, so the LLM applied it to inclusive/"offen für alle" and adult community events (a queer meetup, an adult comedy show both showed up under "für Kinder"). Added an explicit rule to the tag prompt (shared + OpenAI provider copies). Fixed 2 live events by hand.

2026-07-03 — analytics: web vs Telegram channel

Query explorer now shows a channel badge (violet web / sky tg) and filters by it, so web-app traffic is distinguishable from the Telegram bot. Backed by the new query_logs.channel column (tagged at the source in the bot).

2026-07-03 — event-id stability (Stage 0+2) + venue-linking go-forward

Curator keeps the OLDEST event id on merge (Stage 0): pickSurvivor now prefers the earliest-created row over the richest newcomer, so an established id that external references point at (bookmarks, proactive dedup) is never deleted in favour of a re-scrape. Field enrichment onto the survivor is unchanged.
canonical_id_redirects safety net (Stage 2b): before any merge-delete the curator records deleted_id -> survivor_id, so the bot resolves churned ids back to the surviving row. Migration 20260703000000_canonical_id_redirects.sql (additive, service_role-only).
Venue matcher loads inactive venues (venueNormalizer): it was filtering locations to active=true, silently hiding ~158 real venues from matching. Inactive venues now match too — closes part of the unlinked-venue gap going forward (a one-time backfill also linked 356 existing upcoming events).

2026-07-02 — night loop: duplicate creation killed at the provenance layer

Curator upsert now resolves re-ingested events provenance-first (the root cause of the "duplicates came back" class, 179 proven clusters): when the exact (source, source_event_id) pair misses (11/16 sources hash mutable title/time into their ids, so upstream edits shift them), a secondary deterministic key source|normalized-title|date|normalized-venue — time-gated — resolves the event back onto its existing curated row (UPDATE, not a second INSERT).
Provenance self-heal: when one provenance pair maps to MULTIPLE existing curated rows (the legacy duplicate stubs), the richest row survives and the siblings are merged+deleted via the shared mergeCuratedRows — non-fatal, per-cluster, never touches manually_curated rows.
NULL-time twin merges: one-missing-time pairs merge when date+venue match and the title is specific + highly similar (Jaccard ≥0.85/containment with a non-generic shorter side) — hourly tours, markets and cinema screenings are explicitly protected (cinema keeps the strict 5-minute rule, now with the source signal so untagged cinema-source screenings qualify).
Cross-run UPDATEs now union source_events provenance instead of dropping it; model-eval scripts write to events_curated_test (never the live table). +8 curator tests (513 total, tsc clean).
One-time healing of pre-existing duplicates: run POST /api/curator/reset-processed once after this ships, so every raw event re-enters the fixed pipeline and legacy clusters self-heal (~$7 one-off re-enrichment; backup events_curated first).

Added

Junk-row filter at the scrape boundary (filterJunkEvents in src/lib/utils.ts, applied at both filterFutureEvents sites in events/route.ts): drops Eventim ancillary purchase add-ons — parking/coatcheck/shuttle vouchers and fees (Parkplatz, Garderobe, Voucher, Gutschein, Shuttle, …) — so they never enter events_raw / events_curated. One had been served to a bot user as a real event. (The bot also carries a defensive query-time filter.)
Title-aware duplicate cleanup sweep over events_curated (cleanupDuplicates + POST /api/curator/cleanup). Buckets events by (venue, date, time) but merges only when titles are similar (containment / Jaccard / common-prefix / Levenshtein), with an optional embedding-cosine fallback. Deliberately does not merge genuinely distinct events that merely share a slot (cinema screenings on different screens, parallel museum exhibitions), never deletes manually_curated rows, uses a keep-richest survivor and unions source_events. Defaults to a dry run.
Cleanup runs as a final, non-fatal phase of every real curate run so duplicates can no longer accumulate over time.

Changed

Dedupe now uses event-type-aware time matching: non-cinema events can merge across fuzzy start-time offsets up to roughly two hours only when the title match is strong and specific.
Cross-run upsert duplicate matching now follows the same time-aware rules as in-run dedupe, including cinema detection from event tags.

Fixed

Cross-run dedup no longer truncates at PostgREST's 1000-row cap. findCrossRunMatches / findProvenanceMatches now paginate the existing-rows fetch (fetchExistingByDates), so incoming events are compared against the entire curated set for the queried dates instead of only the first 1000. This was the root cause of duplicates accumulating run after run; a one-time sweep removed 979 residual duplicates (~9.8%) from production events_curated.
Cinema/movie dedupe no longer collapses separate screenings at the same venue that start minutes apart; cinema pairs now require near-identical times, unless both sources omit the time.
Generic short recurring events with fuzzy times, such as classes or workshops at the same venue, are protected from accidental merges unless the title evidence is specific enough.
timesWithinThreshold again treats a single missing time as caller-compatible, with stricter missing-time decisions handled in the dedupe/upsert callers.
Curator no longer re-enriches unchanged events every run (major cost fix). The scrape upsert blindly reset events_raw.processed = false on every re-scrape, defeating the curator's unprocessedOnly incremental guard — so ~~9,000 unchanged events were re-grounded (Perplexity) + re-rewritten (OpenAI) + re-embedded daily (~~$7/run; Perplexity spend had risen ~3–4× since Feb–Mar). Added a preserve_processed_on_unchanged BEFORE UPDATE trigger on events_raw (migration 20260529090000) that keeps processed for already-curated rows whose content is unchanged; only genuinely new/changed events are re-curated. Estimated ~80–90% curator cost cut. Reversible (drop the trigger); POST /api/curator/reset-processed forces a full re-curation after a curator-logic change.

Removed

Dead-code purge (grand-refactor W06): removed confirmed zero-caller modules, two destructive @deprecated DELETE helpers, and an orphaned dashboard component island.

Security

Deleted the dead, unauthenticated /api/events/add endpoint (grand-refactor W05) — it accepted event writes with no auth and had no remaining caller.

Internal

First CI pipeline (grand-refactor W02): GitLab CI with lint / test / typecheck / build stages; added the tsc typecheck gate and fixed an orphaned reportHelpers test that referenced helpers living in TagUtils.ts.
Test suite greened + gates made blocking (grand-refactor W23): fixed all 12 failing Vitest tests and 16 tsc errors (incl. rewriting enhance.test to the fail-loud curator contract and deleting a type-fiction perf test); the test and typecheck jobs now block the pipeline (previously advisory).

[0.6.0] - 2026-05-07

Added

Hide-synthetic toggle on /analytics (Phase 1) — distinguishes real bot traffic from QA/test webhooks tagged with is_synthetic=true. Plus a 100/500/1000 row-limit selector and an amber dot on the chip when synthetic is included.
/api/analytics/queries (Phase 2) — paginated rows for the new explorer with cursor-based pagination on (logged_at, id), slim projection that strips flow_details / tool_call_params / similarity_scores from the list path (those load per-row on drawer open).
/api/analytics/queries/[id] — full row including JSONB blobs for the detail view.
/api/analytics/queries/summary — aggregates over the same filter set: total, zero-result %, avg cost, p95 latency, per-bot/tier/intent breakdowns. 60-second in-process TTL cache keyed by filter hash. Backed by a new get_query_log_summary(filter jsonb) Postgres RPC for percentile windows.
/api/analytics/queries/facets?q=<prefix> — autocomplete for user multi-select and bot-handle chips. 5-minute cache, 20-result cap.
QueriesExplorer UI (Phase 3) — TanStack-Table-driven row explorer at the top of /analytics. 9-column compact table, click-to-open detail modal, cursor-paginated load-more, synthetic rows visually differentiated when Include is on.
QueryExplorerFilters — text search (300 ms debounce), date range, tier chips, intent chips, user typeahead, bot-handle chips, synthetic tri-state (Hide/Include/Only), Clear-N-filters control.
AnalyticsHeader with Queries/Reviews tab strip ("Reviews (soon)" placeholder reserves the IA slot for the future user-event-reviews feature).
New indexes: query_logs_bot_date_idx on (bot_handle, logged_at DESC), plus a GIN trigram index on query_text for ilike search.

Changed

/api/analytics route now accepts includeSynthetic and limit query params; default still hides synthetic and caps at 500 rows (lifted from the previous 100).
Realtime stream cap (QueryStream) follows the row-limit selector and filters synthetic INSERT events when the toggle is off.
Release flow in deploy.sh rewritten to use Python instead of sed -i '' for cross-platform parity (Linux/WSL).

Fixed

Cursor pagination strict-validates the decoded cursor (UUID v4 + ISO timestamp + no PostgREST-filter specials like ,/(/)) — defends against .or(...) injection via a tampered cursor param.
/api/analytics/queries/facets?q=... strips PostgREST special chars before interpolating into the .or() expression — same injection class.
Drill-down modal hydrates the slim row with the full /queries/:id payload on open instead of rendering blank flow_details sections.
Filter debounce no longer drops concurrent chip selections (refs to latest value/onChange so the timeout reads fresh state instead of stale closure).
useQueriesExplorer.loadingMore resets at every reload so a mid-fetch filter change doesn't leave the Load-more button permanently disabled.
Dockerfile ships CHANGELOG.md so the public /changelog page can render it.

[0.5.0] - 2026-04-06

Added

Changelog page at /changelog (public, no auth required)
Version footer in layout linking to changelog
Changelog nav link in header
./deploy.sh release [patch|minor|major] command for versioned releases
Semantic versioning with git tags (scraper-vX.Y.Z)
CHANGELOG.md following keepachangelog.com format
Contributor credits (Stefan, Steffi) in changelog

Changed

package.json version bumped to 0.5.0
@tailwindcss/typography plugin for prose rendering

[0.4.0] - 2026-04-05

Added

QuarterbackArena event source — 16th source (Steffi)
Mobile-responsive scraper UI and header (Steffi)
Auto-create missing sources in source_config table (Steffi)
Embedding cosine similarity (0.92 threshold) for cross-run dedup in upsert
Grounding cache for recurring events (saves ~$6.76/curator run)
Parallel LLM enhancement (grounded vs standard events in separate queues)
Title normalization before upsert (strips guillemets, normalizes dashes)
Venue bypass in dedup when titles are very similar (>0.8 Jaccard)
Concurrency limiter (pLimit(5)) for Perplexity API calls
Changelog page at /changelog
Version footer in UI

Fixed

CSS Grid header fix for Viewing selector overlap (Steffi)
Perplexity 429 rate limiting — lowered RPM from 120 to 50
Upsert per-row fallback now uses same conflict key as batch path
Raw events reset processed=false on scrape to trigger re-curation
Null-time dedupe: asymmetric nulls no longer auto-match
Perplexity retry delays increased to 4s/8s/16s

Security

JWT auth with HMAC-SHA256 + alg validation + timing-safe comparison (Stefan)
Logout button in navigation (Stefan)
.mcp.json and partner-api/.env added to .gitignore
partner-api/dist/ removed from git tracking

Removed

6 stale root-level test scripts (771 lines)
partner-api build artifacts from git

[0.3.0] - 2026-01-17

Added

Curator pipeline: fetch → dedupe → merge → venue → tags → enhance → embed → upsert
LLM description enhancement with Perplexity grounding for short descriptions
OpenAI text-embedding-3-small (1536 dims) for event embeddings
Venue normalization against 1,115 known locations
Tag validation and auto-generation
Cross-run dedup via Jaccard similarity + time tolerance
Cinema-aware matching (prevents format variant merges)
Curator run tracking in curator_runs table
Cost tracking (LLM, embedding, Perplexity per run)
Telegram admin notifications for curator errors
CronJobs: daily scrape (5 AM) + daily curator (8 AM)
kinoprogrammLeipzig scraper rewrite for new site structure (Steffi)

Changed

Events pipeline: scraper → events_raw → curator → events_curated
Deduplication uses Union-Find with multi-phase venue grouping

[0.2.0] - 2026-01-10

Added

events_raw table with (source, source_event_id) unique constraint
scrape_runs tracking table
Report UI with analytics and quality metrics at /reports
Source quality metrics and per-source statistics
Source enable/disable management via /api/sources
K8s deployment with CronJobs
Bot integration (/scrape, /sources commands)
14+ event sources: Kreuzer, LeipzigTravel, Werk2, Nato, Urbanite, LeipzigLeben, LeipzigDe, Songkick, regioactive, FrohFroh, kinoprogrammLeipzig, ahoiLeipzig, utConnewitz, cinematheque

[0.1.0] - 2025-11-17

Added

Initial Next.js 15 application with Supabase integration (Stefan)
First event sources: Bandsintown, regioactive, FrohFroh (Stefan)
Basic scraping API with password protection
Cheerio-based HTML parsing
Kreuzer source fixes and kino support (Stefan)
Test mode for debugging (Stefan)
Auto-tagging via keyword matching
Docker deployment to Hetzner k3s cluster