Browse resources

Run your instance / Operate

Configuration

Environment variables, retention, the public address, and upgrading.

knotel is one container: the UI, the OTLP ingest endpoints, the browser script and the MCP server, on port 3000. It keeps spans, logs and web analytics in ClickHouse, and accounts, projects, keys and alerts in a SQLite file in its /data volume. Everything is set with environment variables.

Installing

knotel runs from the public image ghcr.io/kien-ngo/knotel, with Docker Compose (the app and a ClickHouse server side by side) or with plain docker run next to a ClickHouse of your own. Docker and Compose has the Compose file, the commands, and which tag to pin.

Dokploy

Create a Compose service, paste the Compose file from Docker and Compose, and set BETTER_AUTH_SECRET and CLICKHOUSE_PASSWORD in its environment. Remove the ports list and add a domain on port 3000 instead: Dokploy's proxy routes to it, and sends X-Forwarded-Proto and X-Forwarded-Host, so PUBLIC_URL isn't needed there. Add anything else from the table below to the service's environment.

Environment variables

Only the first two are needed to start; everything else has a default or stays off until set.

NameDefaultDescription
BETTER_AUTH_SECRETRequiredSigns sign-in sessions. Generate one with openssl rand -base64 32. Changing it signs everyone out.
CLICKHOUSE_PASSWORDRequiredClickHouse's password. docker-compose.yml gives the same value to the ClickHouse server and to knotel.
PUBLIC_URLFrom the proxy's headersThe address people open knotel at and senders send to, such as https://knotel.example.com. Needed behind a tunnel or a proxy that doesn't send X-Forwarded-Proto and X-Forwarded-Host. Sign-in, snippets and links in alerts use it.
PORT3000The port the server listens on inside the container.
SETUP_TOKENA one-time code in the logThe code /setup asks for before creating the owner account.
RETENTION_DAYS30Days of spans and web analytics a project keeps unless it chooses its own in Settings.
LOG_RETENTION_DAYS7Days of logs, and the most any project keeps them.
STORAGE_LIMIT_GBNoneThe most the data may take on disk until someone sets a limit in Settings; past it, the oldest days are deleted.
STORAGE_CHECK_MINUTES10How often the storage limit is checked.
CLICKHOUSE_URLhttp://clickhouse:8123ClickHouse's HTTP interface. Set by docker-compose.yml; change it to use a server of your own.
CLICKHOUSE_USERknotelThe ClickHouse user. Set by docker-compose.yml.
CLICKHOUSE_DBknotelThe ClickHouse database, created if missing. Set by docker-compose.yml.
DATA_DIR/dataWhere the SQLite file (knotel.db) with accounts, projects and alerts lives.
GEOIPonDownloads the IP → location database once a month. "off" skips it.
KNOTEL_TELEMETRYChosen at setup"off" turns off the anonymous daily report to knotel.dev and locks its switches in Settings. DO_NOT_TRACK=1 does the same.

KNOTEL_IMAGE and HOST_PORT are read by docker-compose.yml itself, not by knotel.

Retention

Spans, logs, metrics, web events and uptime probes are deleted by ClickHouse once they are older than their project keeps them: every row carries its project's number of days, and a table TTL drops the rows past it. Expired data is deleted, not hidden, and there is no job to run for it. A range that reaches further back than retention finds fewer rows, and the range picker strikes it through.

RETENTION_DAYS (30) and LOG_RETENTION_DAYS (7) are the instance's defaults. Web analytics are kept at least 90 days, or longer if the project keeps spans longer, so a month can be compared with the one before.

Per project

Any project can keep less or more than the defaults from Settings → Projects, one menu per kind of data: 1, 3, 7, 14, 30, 60, 90, 180, 365 days. Pick a short one for a staging environment or a noisy internal service, a long one for the production service you actually investigate.

Spans are the main setting; traces, errors, sessions, agent runs and the database page are all read from them. Logs, metrics, web analytics and uptime each follow it until given a number of their own: logs up to LOG_RETENTION_DAYS, metrics exactly, web analytics for at least 90 days, uptime up to 90 days (what a status strip draws). Keeping a year of metrics beside a week of logs is one choice each.

  • It applies from now on. Pages stop showing data past the new window at once. Shortening it also deletes the older data in the background, and the disk space comes back as parts merge. Lengthening it keeps new data longer; what's already stored keeps the expiry it was written with, so the longer window fills in over time.
  • Reads are trimmed at once. A 30-day chart on a project keeping 7 days asks for 7, before the mutation has finished.
  • Only what moved is rewritten. Changing spans rewrites the signals that follow them too; a signal with its own number is left alone.
  • Deleting a project deletes all of its data with it.

A limit on disk

Retention says how long a project keeps its data; the storage limit says how much the machine can hold. Set it in Settings → Storage limit (or with STORAGE_LIMIT_GB until someone does). Every STORAGE_CHECK_MINUTES (10), knotel adds up what spans, logs, web analytics and the trace index take on disk; over the limit, it deletes the oldest days, of every project at once, until usage is under 90% of the limit, so the next check isn't over again straight away.

  • Days go whole. Spans and logs are stored a day per partition, so the oldest day is dropped at once and its space returned immediately. Web analytics and the trace index are cut back to the same day, and their space returns as ClickHouse merges.
  • The limit wins over retention. A project keeping 30 days loses its oldest days first when the disk is full: a full disk would stop every project.
  • Today is never deleted. If today alone is over the limit, the card says so and nothing more is removed: raise the limit, or cut volume with retention and filters.
  • Saving a lower limit trims at once rather than at the next check.
  • Settings wins over the variable. A limit saved in Settings replaces STORAGE_LIMIT_GB, and clearing the field there means no limit at all, not a return to the variable.
  • Leave the disk headroom. The limit counts only knotel's tables in ClickHouse, not the images, container logs, the operating system or the space ClickHouse needs to merge parts. A limit near the free space fills the disk before it ever trims. Aim for about three quarters of what's free once the stack is running: on a 28 GB disk with 21 GB free, 16 GB.
  • The disk itself is guarded too. Whatever the limit, or with none set, the same check reads the free space on ClickHouse's disk and deletes the oldest days to keep 10% of it free, so files the limit doesn't count - an unrotated proxy log, a forgotten ALTER TABLE ... FREEZE under ClickHouse's shadow/ - cost old data rather than the instance. The card shows the disk and says when it, not the limit, is what trims.

Lengthening retention can't bring back what was already deleted: data older than the old number is gone.

Behind a reverse proxy or tunnel

knotel serves plain HTTP on port 3000 and bundles no TLS. Put it behind the reverse proxy or tunnel you already use, and give it the public address in one of two ways:

  • Forwarded headers. A proxy that sends X-Forwarded-Proto and X-Forwarded-Host needs nothing else: each request is treated as addressed to that scheme and host.
  • PUBLIC_URL. Set it when the proxy or tunnel doesn't send them. It wins over the headers.

Sign-in cookies, passkeys, the endpoint in snippets and the links in alerts all use that address, so when it's wrong, sign-in is the first thing to break.

nginx on the same host

Publish knotel on the loopback interface only, with HOST_PORT=127.0.0.1:3000 in .env, and let nginx be the only thing listening publicly. Docker writes its own firewall rules for published ports, so a port published on every interface is reachable even when ufw denies it.

/etc/nginx/sites-available/knotel
server {
    listen 80;
    server_name knotel.example.com;

    # nginx refuses bodies over 1 MB with a 413 by default, and a full OTLP
    # batch can be larger: the exporter drops it and logs at debug level.
    client_max_body_size 32m;

    # Ask AI and the live feed stream their answers and can sit quiet for
    # minutes (a model thinking, a project with nothing new): no buffering,
    # and an hour before nginx gives up on them.
    location ~ ^/api/(chat|stream)$ {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        # MCP streams its responses too.
        proxy_buffering off;
        # nginx's default is 60s; knotel lets a slow query run for 120s.
        proxy_read_timeout 300s;
    }
}

The two timeouts are what the defaults get wrong. nginx closes a response that has sent nothing for 60 seconds, while knotel gives a slow query two minutes and never cuts off Ask AI or the live feed, which stay quiet while a model thinks or nothing new arrives. Whatever proxy or tunnel you use, check its idle timeout the same way: a stream cut short shows up as an Ask AI answer that stops mid-way, or a live feed that reconnects every minute.

Then certbot --nginx -d knotel.example.com --redirect adds TLS, the redirect from HTTP and a renewal timer. Set PUBLIC_URL to the https:// address anyway: it costs nothing and survives a later change of proxy.

OTLP/gRPC through nginx

knotel takes OTLP over gRPC on port 4317, in plaintext. The proxy above doesn't reach it: it only forwards port 3000, over HTTP/1.1. The tidy way to serve gRPC is on the same name and certificate. Every gRPC call is a POST to /package.Service/Method, so the three OTLP services share one path prefix that nginx can send elsewhere. Four extra steps, after certbot has added the 443 server:

  1. Publish the port on loopback only, like 3000: GRPC_HOST_PORT=127.0.0.1:4317 in .env. Published on every interface, the plaintext port is open to the internet whatever the firewall says.
  2. Turn on HTTP/2 in the 443 server. gRPC needs it and nginx serves HTTP/1.1 unless told otherwise. On nginx 1.25.1 and later, add http2 on;. On older versions (Ubuntu 22.04 ships 1.18), add http2 to the listen … ssl lines certbot wrote, e.g. listen 443 ssl http2;. That flag applies to every site on port 443. Browsers switch to HTTP/2 too, and the app works the same over either.
  3. Add the gRPC location to the same server, next to location /:
  4. Set GRPC_PUBLIC_URL to the site's address so the Send data page offers it instead of http://…:4317, then docker compose up -d and nginx -t && systemctl reload nginx.
/etc/nginx/sites-available/knotel (the 443 server)
    listen 443 ssl http2;   # or: http2 on;  (nginx 1.25.1+)

    location /opentelemetry.proto.collector. {
        grpc_pass grpc://127.0.0.1:4317;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        grpc_read_timeout 300s;
    }
.env
GRPC_HOST_PORT=127.0.0.1:4317
GRPC_PUBLIC_URL=https://knotel.example.com

The client_max_body_size already in the server covers gRPC messages as well. The X-Real-IP header does the same job it does for HTTP: it keeps a key's IP allow-list and geolocation working with nginx in front. Senders then use the site's own address, over TLS and with no port:

OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=https://knotel.example.com
OTEL_EXPORTER_OTLP_HEADERS=x-knotel-key=kn_YOUR_KEY

A Collector takes endpoint: knotel.example.com:443 in an otlp_grpc exporter, and leaves out tls: insecure. With Caddy, the same thing is one line in the site block, above the main reverse_proxy: reverse_proxy /opentelemetry.proto.collector.* h2c://127.0.0.1:4317. Caddy serves HTTP/2 already.

The client's address

A key's allowed IP addresses, the fallback visitor id in web analytics and sign-in's rate limit use the client's address. It is read from the first of these that is present:

  1. cf-connecting-ip
  2. X-Real-IP
  3. The first address in X-Forwarded-For
  4. The socket's peer address, with nothing in front
Expose the port only to your proxy
Forwarded headers are only as trustworthy as whatever is in front. If port 3000 is reachable directly, a sender can set any of them and pass an IP allow-list. Publish it on a private interface, or firewall it so only the proxy can connect.

Sign-in

Members sign in with their email and password, and can change the password under Settings → Password. There is no public sign-up: someone who isn't a member is told to ask a teammate to add them.

Passkeys need no configuration. Each member adds one under Settings → Passkeys. The browser only offers them on https or on localhost, and a passkey belongs to the host name it was made on: after moving knotel to another domain, members add new ones.

Geolocation

Backend spans that name their caller's address (client.address, or the older http.client_ip) are located with the free IP Geolocation by DB-IP city database, licensed CC BY 4.0. The server downloads it about 30 seconds after it starts, then checks daily and loads each new month (DB-IP publishes on the 1st). It is kept in the SQLite file, so a restart doesn't download it again. Until the first load finishes, those spans are stored without a location.

GEOIP=off skips the download, for a server with no outbound access or when you'd rather not locate callers. Page views and spans a browser sends are located from the country header a proxy in front adds, if it sends one; they don't need the database.

Backups

There are two things to back up:

  • ClickHouse, which holds spans, logs and web events. Use ClickHouse's own BACKUP, or snapshot the clickhouse-data volume.
  • The SQLite file, knotel.db in /data: accounts, projects, keys, alerts and usage. It is small, and losing it loses the instance, while losing ClickHouse loses only the data.

BACKUP to object storage works without changing ClickHouse's configuration; backing up to a local Disk or File needs a backups disk or allowed path configured first.

# A ClickHouse shell, as knotel's user
docker compose exec clickhouse sh -c 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD"'

-- then, in it
BACKUP DATABASE knotel TO S3('https://s3.example.com/backups/knotel-2026-09-22', 'KEY_ID', 'SECRET');

RESTORE DATABASE knotel FROM S3(…) brings it back. A volume snapshot is consistent when it's atomic (a filesystem or disk snapshot) or taken with ClickHouse stopped.

The SQLite file is in WAL mode, so copying knotel.db alone while the server runs can miss recent writes. Either stop the knotel container for the copy, or write a consistent copy from inside it:

docker compose exec knotel rm -f /data/knotel-backup.db
docker compose exec knotel bun -e \
  "new (require('node:sqlite').DatabaseSync)('/data/knotel.db').exec(\"VACUUM INTO '/data/knotel-backup.db'\")"
docker compose cp knotel:/data/knotel-backup.db ./knotel-backup.db

Upgrading

Pull the new version and recreate the container:

docker compose pull knotel && docker compose up -d

Migrations for both stores, the SQLite file and ClickHouse, run when the new container starts, each recorded so it runs once. There is no separate step. On Dokploy, bump the image tag in the service's compose file and redeploy.

Run a pinned tag rather than a moving one. Every push to main publishes :main and :sha-<commit>; :latest is the current build. With a pinned sha- tag, /api/version tells you which build is running, and rolling back means setting the previous tag and running docker compose up -d again. Migrations don't run backwards, so check what a release changed before rolling back across one.

The changelog is where to check. It lists each build against the commit /api/version reports, says which ones carry a migration, and marks each migration as safe to roll back across or one-way. Most are additive and safe; the ones that rewrite a table are called out, and for those the way back is a backup rather than an older tag.

The browser script is served by the instance, so pages pick up the new one on their own. SDK files copied into your apps only change when you copy them again.