Browse resources

Run your instance / Operate

Troubleshooting

Fixes for common problems, limits, and FAQ.

No data shows up

  • Check the project's Send data page: the key's Last used updates when it's accepted.
  • Look at the ingest request in your browser's network tab or your exporter's logs, and read the response's error. The table below lists each one.
  • A 200 with a partialSuccess saying Collection is paused for this project in knotel Settings. Nothing was stored. means the project's collection switch is off. Turn it on in Settings → Projects. A partialSuccess starting Dropped … names spans refused for unusable ids or timestamps, or past the 5,000-span limit.
  • Browser spans are sent about 2 seconds after they happen, and when the tab is hidden. Wait a moment, then refresh.
  • Pages open on the last 24 hours. Test spans with old timestamps fall outside it; see Spans land at the wrong time.
  • A Cloudflare Worker you trace needs the nodejs_compat flag, and its handler must be the one wrapped by instrument() and exported as default.
  • The service's boot line says tracing is on but nothing arrives: exporters swallow network and auth errors. Run it once with OTEL_LOG_LEVEL=debug, and check the key with the empty-batch request.

Ingest responses

StatusMessageFix
401Missing API key. Send it in the x-knotel-key header.Send the key in x-knotel-key or Authorization: Bearer. A service still sending x-flaretrace-key gets this too: that name is no longer read, though ft_ keys themselves still work.
401Invalid API key.The key is wrong, revoked, or belongs to a deleted project.
403This key doesn't accept data from here (…)The request came from a domain or IP the key doesn't allow. The message names what it saw; add it to the key's allowed sources on the Send data page.
413Payload over 5 MB. Send smaller batches.Lower the exporter's batch size.
415Send OTLP/HTTP as JSON or protobuf to /v1/traces. OTLP/gRPC goes to port 4317.The exporter's protocol doesn't match the port: http/json or http/protobuf to the site's URL, grpc to port 4317.
400Body is not valid JSON. / Body is not valid OTLP protobuf.The content-type doesn't match the body, or the body is cut off.
503Could not store spans. Retry later.ClickHouse refused the insert. See below.

ClickHouse is unreachable

knotel connects to ClickHouse on the first request after it starts and applies its migrations. Until that works, every page and ingest request fails with a server error, the container's health check fails, and the log (docker compose logs knotel) shows ClickHouse's own error. Nothing is lost: the next request tries again.

  • Connection refused or timed out: ClickHouse isn't up yet or CLICKHOUSE_URL is wrong. Compose waits for its health check, so on the bundled stack check docker compose ps clickhouse and its log.
  • Authentication failed: CLICKHOUSE_USER or CLICKHOUSE_PASSWORD doesn't match the server. In the bundled stack both containers read them from .env, so after changing either, recreate both with docker compose up -d. With your own ClickHouse, check the user and password against that server.
  • CLICKHOUSE_DB must be a plain identifier: the database name may only contain letters, digits and underscores.
  • Could not store spans. Retry later. (503) once it was working: ClickHouse refused an insert, most often because its disk is full. The log has span insert failed (or log insert failed) with the reason. OTLP exporters retry a 503, so a short outage costs a delay, not data.

Some spans never appear

The service reports its HTTP spans, but not its database, Redis or log records - or reports nothing but a few outgoing calls. The instrumentation says it's enabled and patched nothing, because the library loaded before the SDK started. Check, in order:

  • The tracing file is preloaded with --import or --require, or is the first import of the entry file.
  • An ESM app registers @opentelemetry/instrumentation/hook.mjs before it imports anything instrumented.
  • Nothing imported by the tracing file itself - a shared barrel, your logger - pulls in the library early.
  • The library isn't inlined by a bundler, and the server isn't Bun.serve, which bypasses node:http.

Production rollout has the details of each.

Sign-in fails

  • No passkey prompt: browsers offer passkeys only on https or localhost, so open knotel through your https address, not http://server-ip:3000. A passkey made on another host name doesn't work here; add a new one in Settings → Passkeys.
  • Everyone was signed out: BETTER_AUTH_SECRET changed.

Spans land at the wrong time

Spans are stored at the time the sender says they happened. A sender whose clock is behind puts them in the past, outside the range on screen; one further behind than the project's retention sends spans that ClickHouse deletes as already expired. A clock that's ahead puts them after now, where no range finds them until time catches up. Keep senders and the knotel host on NTP; date -u on each shows the difference.

Browser and backend traces aren't linked

  • The backend's host must be your page's origin or be listed in data-propagate.
  • Cross-origin APIs must allow the traceparent header in CORS. If they don't, the browser blocks the request itself, so check the console.
  • The backend must read traceparent. The Workers SDK and OpenTelemetry SDKs do.

A Python service traces nothing

Look for 415 in the service's own log. The Python SDK ships only a protobuf exporter and ingest accepts protobuf, so a 415 means the content type isn't reaching knotel, often because a proxy in front rewrites or rejects it. A rejected batch is dropped and the service carries on, which is why tracing can look healthy while nothing is stored. See OpenTelemetry.

Cloudflare Worker spans show 0 ms

Cloudflare Workers only advance the clock during I/O, so CPU-only work in an app you trace there has no measurable duration. Spans around fetches, database calls and other I/O are accurate.

Numbers lag behind

Spans can be queried as soon as ingest answers 200. Dashboard results are cached so everyone looking at the same range shares one: for 30 seconds (15m), a minute (1h), 2 minutes (6h), 5 minutes (24h), 15 minutes (7d) and 30 minutes (30d). Switch to a short range, or use Refresh, for live numbers.

A revoked key still works

The server keeps recent key lookups in memory for a minute, so a key used just before it was revoked can be accepted for up to a minute more.

Limits

LimitValue
Ingest request body5 MB
Spans per ingest request5,000 (extra spans are rejected)
Log records per ingest request10,000
Attribute string length8,000 characters
Query text4,000 characters
Status message2,000 characters
Span name500 characters
Service name200 characters
RetentionRETENTION_DAYS (30) or a project's own, per signal; logs follow LOG_RETENTION_DAYS (7) until set
StorageThe ClickHouse server's disk

FAQ

Does it do logs or metrics? Traces and logs. There's no metrics endpoint; many questions metrics answer (request rates, error rates, percentiles) come from traces.

Does it sample? No. Every span sent is stored. Sample in your SDK or Collector if you need to.

Can several environments share an instance? Yes. Use a project per environment, or one project with a deployment.environment.name resource attribute to filter on.