Run your instance / Operate
Troubleshooting
Fixes for common problems, limits, and FAQ.
No data shows up
- Check the project's Send data page: the key's Last used updates when it's accepted.
- Look at the ingest request in your browser's network tab or your exporter's logs, and read the response's
error. The table below lists each one. - A
200with apartialSuccesssaying Collection is paused for this project in knotel Settings. Nothing was stored. means the project's collection switch is off. Turn it on in Settings → Projects. ApartialSuccessstarting Dropped … names spans refused for unusable ids or timestamps, or past the 5,000-span limit. - Browser spans are sent about 2 seconds after they happen, and when the tab is hidden. Wait a moment, then refresh.
- Pages open on the last 24 hours. Test spans with old timestamps fall outside it; see Spans land at the wrong time.
- A Cloudflare Worker you trace needs the
nodejs_compatflag, and its handler must be the one wrapped byinstrument()and exported as default. - The service's boot line says tracing is on but nothing arrives: exporters swallow network and auth errors. Run it once with
OTEL_LOG_LEVEL=debug, and check the key with the empty-batch request.
Ingest responses
| Status | Message | Fix |
|---|---|---|
| 401 | Missing API key. Send it in the x-knotel-key header. | Send the key in x-knotel-key or Authorization: Bearer. A service still sending x-flaretrace-key gets this too: that name is no longer read, though ft_ keys themselves still work. |
| 401 | Invalid API key. | The key is wrong, revoked, or belongs to a deleted project. |
| 403 | This key doesn't accept data from here (…) | The request came from a domain or IP the key doesn't allow. The message names what it saw; add it to the key's allowed sources on the Send data page. |
| 413 | Payload over 5 MB. Send smaller batches. | Lower the exporter's batch size. |
| 415 | Send OTLP/HTTP as JSON or protobuf to /v1/traces. OTLP/gRPC goes to port 4317. | The exporter's protocol doesn't match the port: http/json or http/protobuf to the site's URL, grpc to port 4317. |
| 400 | Body is not valid JSON. / Body is not valid OTLP protobuf. | The content-type doesn't match the body, or the body is cut off. |
| 503 | Could not store spans. Retry later. | ClickHouse refused the insert. See below. |
ClickHouse is unreachable
knotel connects to ClickHouse on the first request after it starts and applies its migrations. Until that works, every page and ingest request fails with a server error, the container's health check fails, and the log (docker compose logs knotel) shows ClickHouse's own error. Nothing is lost: the next request tries again.
- Connection refused or timed out: ClickHouse isn't up yet or
CLICKHOUSE_URLis wrong. Compose waits for its health check, so on the bundled stack checkdocker compose ps clickhouseand its log. Authentication failed:CLICKHOUSE_USERorCLICKHOUSE_PASSWORDdoesn't match the server. In the bundled stack both containers read them from.env, so after changing either, recreate both withdocker compose up -d. With your own ClickHouse, check the user and password against that server.CLICKHOUSE_DB must be a plain identifier: the database name may only contain letters, digits and underscores.Could not store spans. Retry later.(503) once it was working: ClickHouse refused an insert, most often because its disk is full. The log hasspan insert failed(orlog insert failed) with the reason. OTLP exporters retry a 503, so a short outage costs a delay, not data.
Some spans never appear
The service reports its HTTP spans, but not its database, Redis or log records - or reports nothing but a few outgoing calls. The instrumentation says it's enabled and patched nothing, because the library loaded before the SDK started. Check, in order:
- The tracing file is preloaded with
--importor--require, or is the first import of the entry file. - An ESM app registers
@opentelemetry/instrumentation/hook.mjsbefore it imports anything instrumented. - Nothing imported by the tracing file itself - a shared barrel, your logger - pulls in the library early.
- The library isn't inlined by a bundler, and the server isn't
Bun.serve, which bypassesnode:http.
Production rollout has the details of each.
Sign-in fails
- No passkey prompt: browsers offer passkeys only on https or
localhost, so open knotel through your https address, nothttp://server-ip:3000. A passkey made on another host name doesn't work here; add a new one in Settings → Passkeys. - Everyone was signed out:
BETTER_AUTH_SECRETchanged.
Spans land at the wrong time
Spans are stored at the time the sender says they happened. A sender whose clock is behind puts them in the past, outside the range on screen; one further behind than the project's retention sends spans that ClickHouse deletes as already expired. A clock that's ahead puts them after now, where no range finds them until time catches up. Keep senders and the knotel host on NTP; date -u on each shows the difference.
Browser and backend traces aren't linked
- The backend's host must be your page's origin or be listed in
data-propagate. - Cross-origin APIs must allow the
traceparentheader in CORS. If they don't, the browser blocks the request itself, so check the console. - The backend must read
traceparent. The Workers SDK and OpenTelemetry SDKs do.
A Python service traces nothing
Look for 415 in the service's own log. The Python SDK ships only a protobuf exporter and ingest accepts protobuf, so a 415 means the content type isn't reaching knotel, often because a proxy in front rewrites or rejects it. A rejected batch is dropped and the service carries on, which is why tracing can look healthy while nothing is stored. See OpenTelemetry.
Cloudflare Worker spans show 0 ms
Cloudflare Workers only advance the clock during I/O, so CPU-only work in an app you trace there has no measurable duration. Spans around fetches, database calls and other I/O are accurate.
Numbers lag behind
Spans can be queried as soon as ingest answers 200. Dashboard results are cached so everyone looking at the same range shares one: for 30 seconds (15m), a minute (1h), 2 minutes (6h), 5 minutes (24h), 15 minutes (7d) and 30 minutes (30d). Switch to a short range, or use Refresh, for live numbers.
A revoked key still works
The server keeps recent key lookups in memory for a minute, so a key used just before it was revoked can be accepted for up to a minute more.
Limits
| Limit | Value |
|---|---|
| Ingest request body | 5 MB |
| Spans per ingest request | 5,000 (extra spans are rejected) |
| Log records per ingest request | 10,000 |
| Attribute string length | 8,000 characters |
| Query text | 4,000 characters |
| Status message | 2,000 characters |
| Span name | 500 characters |
| Service name | 200 characters |
| Retention | RETENTION_DAYS (30) or a project's own, per signal; logs follow LOG_RETENTION_DAYS (7) until set |
| Storage | The ClickHouse server's disk |
FAQ
Does it do logs or metrics? Traces and logs. There's no metrics endpoint; many questions metrics answer (request rates, error rates, percentiles) come from traces.
Does it sample? No. Every span sent is stored. Sample in your SDK or Collector if you need to.
Can several environments share an instance? Yes. Use a project per environment, or one project with a deployment.environment.name resource attribute to filter on.