Browse resources

Send data

Send metrics

Take OpenTelemetry metrics at /v1/metrics: counters, gauges and histograms, charted per service, pod or route.

knotel takes OpenTelemetry metrics at /v1/metrics, in JSON or protobuf, with the same ingest key as traces. Every point type OTLP has is kept: gauges, counters (monotonic sums), up-down counters, histograms, exponential histograms and summaries.

Sending

Point any OTel metrics exporter, or a Collector's otlphttp exporter, at the endpoint. The key goes in the same header traces use.

OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://knotel.example.com/v1/metrics
OTEL_EXPORTER_OTLP_METRICS_HEADERS=x-knotel-key=kn_...

Either temporality works. Cumulative (the SDKs' default) and delta counters chart the same; there is nothing to configure on either side.

Python

from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

reader = PeriodicExportingMetricReader(OTLPMetricExporter(), export_interval_millis=60_000)
metrics.set_meter_provider(MeterProvider(metric_readers=[reader]))

How counters are kept

A cumulative counter sends a running total. knotel works out what each point added since the one before as it arrives, and stores that increase beside the total, so a rate is a sum and a month of it is read from hourly rows rather than every point.

  • A process restart (a new start time, or a total that went down) counts everything the new process has counted, not a negative step.
  • An exporter's retry is recognised by its timestamp and stored once, so a flaky network never doubles a rate.
  • A counter that was already running when it was first pointed here counts from its second point: an hour of history arriving at once would otherwise draw as a spike.
  • knotel restarting loses nothing: each series picks up from its last stored point.
The running totals are worked out in the knotel process, so run one knotel per ClickHouse, as the Compose file does. Two behind one address would each see half of a series' points.

Reading

The Metrics page lists what a project sent and charts one metric at a time, split into lines by any point attribute, service.name, or a resource attribute as resource.<key> (resource.k8s.pod.name, resource.host.name). What a chart can show depends on the metric:

MetricCharted as
Gauge, up-down counteraverage, max, min, or sum across series
Counterper second, or per bucket
Histogramp50/p90/p95/p99 of each bucket's recordings, mean, per second
Summarythe sender's own quantiles, averaged; mean; per second

Histogram percentiles are of what was recorded in each bucket, not since the process started, and are as precise as the histogram's buckets: exponential histograms, which the Node and Java SDKs can send, give the closest answers. Over MCP, list_metrics and query_metric ask the same questions.

Retention

Metrics are kept as long as the project keeps spans. A series is stored once and its points carry only their numbers, so a metric costs a few bytes a point on disk.