Architecture and trade-offs

knotel is a tracing backend, dashboard and MCP server in one container, with ClickHouse beside it for spans, logs and web analytics and a SQLite file for everything else. This page explains how that works, why it's built this way, and where the design stops being the right fit.

The short version

• Two containers. Ingest, queries, the web app, auth, MCP and the alert checks are one Bun process; ClickHouse is the other. docker compose up starts both.
• Your machine, your data. Spans are written to the ClickHouse you run and nowhere else.
• One columnar store. ClickHouse keeps every span, compressed by column, for as long as its project keeps it. There is no cache of recent spans in front and no slower archive behind: a 30-day chart reads the same table as a 15-minute one.
• Aggregates are computed from raw spans. ClickHouse scans and aggregates columns fast enough that there are no rollups and no sampling needed for capacity.
• Priced as a machine. No per-span or per-seat bill: a small VM with 2-4 vCPUs and 4-8 GB of memory handles millions of spans a day.
• OpenTelemetry-native. Spans and logs arrive as OTLP over HTTP from the bundled browser script, the Workers SDK, or any OTel SDK.

The pieces

Life of a span

1. Captured. The browser script patches fetch, listens for errors and reads the navigation timing API; the Workers SDK, for apps on Cloudflare Workers, wraps the handler and uses AsyncLocalStorage to know which request an outgoing fetch belongs to. Both add a W3C traceparent header so the next service continues the same trace.
2. Batched. The browser sends up to 50 spans per request (sooner when the tab hides, with keepalive). A Worker sends a request's spans after the response, with ctx.waitUntil, so users never wait on telemetry.
3. Authenticated. The ingest key is hashed and looked up in memory, then in a cache table, then in the project tables, all in the SQLite file. Most requests never leave memory to authenticate.
4. Parsed. OTLP JSON and protobuf are normalised the same way: hex or base64 ids, numeric or named enums, nanosecond strings. Both read the same OTLP fields, so neither encoding carries less. Well-known attributes (HTTP method, route, status, URL, database system, operation, collection, query) become columns, as do the instrumentation scope, trace state and flags, and the counts of what the sender's SDK dropped; everything else stays as JSON.
5. Filtered. Project filters drop scanner probes and other noise in memory, before any write.
6. Stored once. The batch is one insert into ClickHouse's spans table, each row carrying its project's retention in days. Inserts are asynchronous on the server side, which gathers many small batches into one part, but ingest waits for the flush: a 200 means stored.
7. Metered. Requests, spans, bytes and what was dropped are added to daily usage counters in SQLite after the response.

Reading: one store

Every chart, table and MCP answer comes from one query module, which trims the requested range to what the project still keeps and asks ClickHouse. The spans table is sorted by project, service and time, so one project's window is a narrow range of each column, read in parallel.

Entry spans

Request counts, error rates and latency use only server spans and trace roots, so a request with twenty database calls counts once, with its end-to-end latency.

Percentiles

p50, p95 and p99 come from ClickHouse's t-digest quantiles: approximate, but close enough to read trends from, and computed over every span in the range rather than a sample.

The service map is a self-join

Joining each span to its parent finds every place where the parent belongs to a different service or project: that's an edge. Database and external-host edges come from client spans that no instrumented service answered. No extra instrumentation or topology data is needed.

Home signals are computed on read

The Needs attention list compares each service's current window with the one before (error rate, p95, traffic) when you open Home. There's no background job behind it. Alerts are the scheduled counterpart: a threshold on one of the same numbers, checked every minute by a timer in the knotel process.

Caching

• Credentials: memory (60 s) → a cache table in SQLite (5 min) → the key tables. Revoking deletes the cache entry at once.
• Dashboards: every query is cached in memory, then in the cache table, for 30 s (15m) up to 30 min (30d), and everyone looking at the same range shares one result. Past that, the old result is still served while a new one loads in the background, so a viewer never waits on a query just because the cache turned over.
• Filters and retention: held in memory and refreshed from SQLite, so ingest reads neither per request.

Auth and MCP

Better Auth handles email and password and passkeys, with sessions in SQLite. There's no public sign-up: the first account claims the instance with a setup code, and admins invite everyone else. The public address comes from PUBLIC_URL or the proxy's X-Forwarded-Proto and X-Forwarded-Host.

The MCP server is stateless Streamable HTTP: each request builds the tool set, answers, and exits. Tools call the same query module as the dashboard, so the AI and the UI never disagree.

Why ClickHouse

• Compression. Spans repeat themselves: the same resource JSON, service names and routes on every row. Stored by column and sorted by service, that compresses around 10× or better, so months of history fit on an ordinary disk.
• Aggregation over raw data. Counts, error rates and percentiles over millions of spans come straight from the table, so there's nothing to precompute and no view that only holds summaries.
• Retention as a TTL. Each row expires on its own, and ClickHouse drops expired data as it merges. There is no cleanup job to schedule or fall behind.
• SQLite for the rest. Accounts, projects, keys and alerts are small and relational, and need no server of their own: one file in /data.

Trade-offs and limits

Compared with the alternatives

Where it goes next

• Alerts offered from the page you're on, a visual query builder with heatmaps, and OAuth for MCP.

Data model

ClickHouse's spans table holds one row per span: ids, service, name, kind, timings, status, the promoted HTTP and database attributes, the instrumentation scope, trace state and flags, the sender's dropped counts, when ingest received the span, its retention, and attributes, resource, events and links as JSON text. It is partitioned by day and sorted by project, service and time.

Code layout