OpenTelemetry

Anything that speaks OpenTelemetry can send traces to knotel: Node.js, Python, Go and Java services, and the OpenTelemetry Collector. Auto-instrumentation libraries give you database spans for MongoDB, Postgres, Redis and more without code changes.

Endpoint

Environment variables

SDKs that read the standard variables need only these:

OTEL_SERVICE_NAME=orders-api
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://YOUR-INSTANCE/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS=x-knotel-key=kn_YOUR_KEY

Node.js

@opentelemetry/exporter-trace-otlp-http sends JSON, so it works directly. With auto-instrumentations you get HTTP, Express, MongoDB, pg, Redis and other spans automatically.

// instrumentation.ts, loaded before your app (node --import ./instrumentation.js)
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";

new NodeSDK({
  serviceName: "orders-api",
  traceExporter: new OTLPTraceExporter({
    url: "https://YOUR-INSTANCE/v1/traces",
    headers: { "x-knotel-key": process.env.KNOTEL_KEY },
  }),
  instrumentations: [getNodeAutoInstrumentations()],
}).start();

Other languages

Python's and Go's HTTP exporters send protobuf, which ingest accepts directly — no gateway needed. For batching or sampling you can still put an OpenTelemetry Collector in front and have it forward to knotel. The Collector batches, and can sample or drop noisy spans before they cost you anything.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlphttp/knotel:
    traces_endpoint: https://YOUR-INSTANCE/v1/traces
    compression: none
    headers:
      x-knotel-key: kn_YOUR_KEY

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/knotel]

Attributes knotel understands

Every attribute is stored and searchable. These are also indexed as columns for fast filtering and power the service map. Older semantic convention names are accepted too.

CPU and memory

knotel takes traces and logs, and there is no /v1/metrics: an OpenTelemetry host-metrics or runtime-metrics exporter has nowhere to send. Measure the process across a request instead and put the numbers on its root span, where they are filterable like any other attribute.

In Node, read the counters in the HTTP instrumentation's hooks. They run while the span is still open, which a res.on("finish") listener does not: the instrumentation registers its own listener first, ends the span, and your attributes land on a span that has already closed.

const cpuAt = new WeakMap<object, NodeJS.CpuUsage>();

const instrumentations = [
  getNodeAutoInstrumentations({
    "@opentelemetry/instrumentation-http": {
      requestHook: (span) => {
        cpuAt.set(span, process.cpuUsage());
      },
      responseHook: (span) => {
        const at = cpuAt.get(span);
        if (!at) return;
        const cpu = process.cpuUsage(at); // microseconds since `at`
        const mem = process.memoryUsage();
        span.setAttributes({
          "process.cpu.user_us": cpu.user,
          "process.cpu.system_us": cpu.system,
          "process.memory.rss_mb": Math.round(mem.rss / 1e6),
          "process.memory.heap_used_mb": Math.round(mem.heapUsed / 1e6),
        });
      },
    },
  }),
];

In Python the ordering is easier: a FastAPI middleware runs inside the server span, so the span is still open after call_next returns.

import resource
from opentelemetry import trace

@app.middleware("http")
async def cpu_and_memory(request, call_next):
    before = resource.getrusage(resource.RUSAGE_SELF)
    response = await call_next(request)
    after = resource.getrusage(resource.RUSAGE_SELF)
    span = trace.get_current_span()
    span.set_attribute(
        "process.cpu.user_us", round((after.ru_utime - before.ru_utime) * 1e6)
    )
    span.set_attribute(
        "process.cpu.system_us", round((after.ru_stime - before.ru_stime) * 1e6)
    )
    # ru_maxrss is peak RSS: kilobytes on Linux, bytes on macOS.
    span.set_attribute("process.memory.rss_mb", after.ru_maxrss // 1024)
    return response

Those names are yours — knotel special-cases none of them, and any attribute is stored and searchable. What you get for free is filtering with numeric operators, so process.memory.rss_mb > 400 narrows the Traces page like a column does, and outlier attribution with no configuration at all: numbers that rarely repeat are compared as quartile ranges, so "the slowest 1% ran with user CPU in the top quartile" surfaces on its own.

Sampling

knotel samples nothing itself: every span it receives is stored. When your SDK samples, tell knotel the rate and counts are scaled back up — a service sampling at 2% otherwise reads as fifty times quieter than it is. Error rates and percentiles survive sampling on their own; counts do not.

OpenTelemetry's consistent probability samplers write their rate into the W3C tracestate header as ot=th:<threshold>, and knotel reads it with no setup at all. That covers the Collector's probabilistic_sampler processor and the SDKs' consistent samplers, where the rate rides on every span of a sampled trace.

Plain TraceIdRatioBased is not one of them: it drops spans and records nothing about it. For that sampler, and for sampling you do yourself, set one of these as a span attribute, or on the resource for a whole service:

A span attribute wins over tracestate, which wins over a rate on the resource: each is more specific about this span than the next. Undeclared and unpropagated means 1: one span counts as one request. Nothing else is inferred, because guessing a rate would invent traffic that never happened. Where counts are scaled, the dashboard says how many spans are behind the estimate.

Responses

Ids may be hex or base64, kinds and status codes numbers or enum names, and timestamps nanoseconds as numbers or strings. Spans without a valid trace id, span id or start time are rejected. Long strings are trimmed (attributes at 8,000 characters, query text at 4,000). CORS is open, so browsers can post directly. See limits.