Browse resources

Send data / Logs and metrics

Send logs

Take OpenTelemetry logs at /v1/logs, joined to the trace that wrote them.

knotel takes OpenTelemetry logs at /v1/logs, in JSON or protobuf, with the same ingest key as traces. A record that carries a trace id is joined to that trace: its lines appear under the waterfall on the trace page, and get_trace returns them beside the spans.

Sending

Point any OTel logs exporter at the endpoint. The key goes in the same header traces use.

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://knotel.example.com/v1/logs
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-knotel-key=kn_...

To join a line to its trace, log inside an active span. Every OTel logging bridge copies the current trace and span id onto the record for you; nothing joins a line written outside a span.

Cloudflare Workers

The Workers SDK can record console output as logs. It is off by default, because a chatty Worker writes far more lines than spans:

export default instrument(handler, (env) => ({
  endpoint: "https://knotel.example.com",
  key: env.KNOTEL_KEY,
  service: "api",
  logs: true,          // console.log/info/warn/error; add { level: "debug" } for debug
}));

Lines are stamped with the active span's ids, so they land on the right trace. Workers only advance the clock on I/O, so several lines can share a timestamp; ties get a microsecond step each so their order survives.

Python

from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

provider = LoggerProvider()
provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
logging.getLogger("my_app").addHandler(LoggingHandler(logger_provider=provider))
Attach the handler to your own logger, not the root one. On the root logger at DEBUG it captures urllib3's line about the exporter's own POST, which it then exports, which logs another line: the process spins and never stops sending.

Severity

OTel numbers severity 1-24 in ranges of four. When a sender gives only a name, knotel infers the number from it, so warning, WARN and W all filter as WARN; a name nothing recognises stores as 0 and only matches "all levels".

RangeLevel
1-4TRACE
5-8DEBUG
9-12INFO
13-16WARN
17-20ERROR
21-24FATAL

Asking for WARN and above filters on the stored number, so it's the same query whichever name the sender used.

Retention

Logs are kept for LOG_RETENTION_DAYS (7 by default). That is shorter than the span default on purpose: a service writes far more lines than spans. A project whose span retention is shorter keeps its lines for that long instead, and a project can give logs a number of its own - shorter, or longer than the default - in Settings.

  • Records live in their own ClickHouse table, and a TTL deletes each line once it is older than its project's log retention. Expired lines are deleted, not hidden.
  • A range reaching further back than retention simply finds fewer lines. The Logs page says how many it found.

Raise LOG_RETENTION_DAYS if you have the disk for it; ClickHouse compresses log bodies and attributes well, and the Usage page shows how much the logs table takes on disk.

Reading

  • Logs page: newest first, filtered by service, level and a substring of the message. Click a line for its attributes and a link to its trace.
  • Trace page: a trace's own lines under the waterfall, in the order they were written. Selecting a span dims the lines that weren't written during it.
  • MCP: search_logs for the same filters, and get_trace, which returns a trace's lines with its spans.

Log records are never scaled by a sample rate. Head sampling decides which spans are exported; logging SDKs emit every record regardless, so one row is one line.