Production rollout

Getting one span into knotel takes five minutes. Keeping every service reporting through deploys, renames, sampling changes and a second tracing tool takes a few more decisions. This page collects them. It comes from rolling knotel out across a dozen production services — NestJS, Fastify, Hono on Node and on Bun, FastAPI, a bundled MCP server, two React apps and a marketing site — and nearly every rule here is there because breaking it once lost data without an error.

One configuration contract

Give every service the same two variables and the same rules for them. When every repository reads them the same way, a rename or a key rotation is a search, not an investigation.

KNOTEL_ENDPOINT=https://YOUR-INSTANCE   # the base URL; code appends /v1/traces and /v1/logs
KNOTEL_KEY=kn_…                    # an ingest key: write-only, one per environment

• Off unless both are set. Local development and tests then never export, and nobody needs a fake key to run the suite.
• Strip trailing slashes from the endpoint. A hand-pasted https://YOUR-INSTANCE/ otherwise becomes //v1/traces, which is a 404.
• Never let tracing stop the service. Wrap setup in a try/catch, import the SDK lazily inside it, and log what went wrong. A malformed endpoint that the exporter rejects at construction should cost you traces, not the deploy.
• Print one boot line, either [knotel] tracing on → https://YOUR-INSTANCE as orders-api or [knotel] tracing OFF (KNOTEL_KEY not set), and make the second a warning in production.
• Use one project and one key per environment. Staging traffic then can't dilute production's numbers, and revoking a leaked staging key touches nothing else.

export function knotelConfig() {
  const endpoint = (process.env.KNOTEL_ENDPOINT ?? "").trim().replace(/\/+$/, "");
  const key = (process.env.KNOTEL_KEY ?? "").trim();
  if (!endpoint || !key) {
    const log = process.env.ENVIRONMENT === "production" ? console.warn : console.info;
    log("[knotel] tracing OFF (KNOTEL_ENDPOINT or KNOTEL_KEY not set)");
    return null;
  }
  return { traces: `${endpoint}/v1/traces`, logs: `${endpoint}/v1/logs`, key };
}

Check the key before you deploy

The boot line proves the variables are set, not that anything arrives. An empty batch tests the key, the header and the network path in one request: 200 means ingest accepted it, 401 means the key or header is wrong, 403 means the key doesn't allow this source.

curl -i -X POST "$KNOTEL_ENDPOINT/v1/traces" \
  -H 'content-type: application/json' \
  -H "x-knotel-key: $KNOTEL_KEY" \
  -d '{"resourceSpans":[]}'

For the whole pipeline, keep a smoke script that sends one span through the real exporter with the SDK's diagnostics turned up, then flushes. A wrong key or a DNS failure is invisible without DiagLogLevel.DEBUG (or OTEL_LOG_LEVEL=debug).

import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import { BasicTracerProvider, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);
const provider = new BasicTracerProvider({
  resource: resourceFromAttributes({ "service.name": "smoke-test" }),
  spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({
    url: `${process.env.KNOTEL_ENDPOINT}/v1/traces`,
    headers: { "x-knotel-key": process.env.KNOTEL_KEY! },
  }))],
});
provider.getTracer("smoke").startSpan("GET /smoke-test").end();
await provider.forceFlush();
await provider.shutdown();

Start the SDK before anything else loads

Node instrumentation works by patching a module as it's first loaded. Anything loaded before the SDK starts is never patched, and nothing says so: the instrumentation reports itself enabled and produces no spans. Every way this went wrong looked exactly like a misconfigured endpoint.

• Preload it. node --import ./dist/tracing.js dist/main.js (ESM) or --require (CommonJS), in the start script, the Dockerfile's CMD and your process manager's node_args alike. The preload runs before your app, so it loads dotenv/config itself if you use it.
• Or make it the first import of main.ts, and import it by its own path. A shared package's barrel file that also re-exports a Redis cache loads ioredis first, and every Redis call then bypasses tracing.
• ESM needs the loader hook. Without it, instrumentation of imported modules registers and patches nothing:

import { register } from "node:module";
register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);

// …start the SDK, then load the app dynamically so fastify, pg and pino
// are imported after the patches are in place:
await startTracing();
await import("./app.js");

• Load your logger after the SDK. A pino instance created before PinoInstrumentation registers keeps logging and never gets trace ids or exports a record, while spans keep flowing. If the tracing module needs to log, import the logger dynamically after sdk.start().
• Bundled code can't be patched. A bundler that inlines mongoose leaves nothing for require to intercept. Instrumentations built on diagnostics channels, such as undici, still work; for the rest, mark the library external or add spans by hand.
• Bun serves without node:http, so HttpInstrumentation produces no server span, and Bun's fetch isn't undici. See Bun for the few lines that replace both.

Name what's sending

knotel promotes four resource attributes to columns, so every page can filter and compare by them:

On Kubernetes, add the pod, namespace and node too — one leaking pod and a fleet-wide drift look identical without them. The Kubernetes section has the downward-API YAML.

Keep the noise out

Most span volume in a default Node setup is spans nobody reads. Drop them where they're made — creating no span is cheaper than sampling one away, and much cheaper than storing it.

const knotelHost = new URL(config.traces).host;

getNodeAutoInstrumentations({
  // File, DNS and socket spans were most of the volume and none of the answers.
  "@opentelemetry/instrumentation-fs": { enabled: false },
  "@opentelemetry/instrumentation-dns": { enabled: false },
  "@opentelemetry/instrumentation-net": { enabled: false },
  "@opentelemetry/instrumentation-http": {
    // Probes: no span at all.
    ignoreIncomingRequestHook: (req) => /^\/health(\/|$)/.test(req.url ?? ""),
    // The exporter's own posts, or each export traces the next one.
    ignoreOutgoingRequestHook: (opts) => opts.hostname === knotelHost,
  },
  // A query outside any request would otherwise start its own trace
  // and be counted as a request.
  "@opentelemetry/instrumentation-mongoose": { requireParentSpan: true },
});

// NodeSDK exports metrics and logs to localhost:4318 by default, and fails
// there every minute. knotel has no /v1/metrics.
process.env.OTEL_METRICS_EXPORTER ??= "none";
process.env.OTEL_LOGS_EXPORTER ??= "none";

• Match health paths exactly: /health and /health/…, not anything starting with /health.
• Anything that still gets through — scanner probes, crawlers, uptime checks — can be dropped at ingest with filters, by rule, with no redeploy.
• Python's BatchSpanProcessor logs a full traceback whenever a load balancer closes an idle pooled connection, though the spans arrive on the retry. Raise opentelemetry.sdk.trace.export to CRITICAL if it floods your logs.

Sample by endpoint, and say so

One ratio for a whole service keeps too much of the one endpoint that's 90% of the traffic and too little of everything else. What worked was a small policy: probes at 0, the hot endpoint at 1%, the rest at 100%, and everything at 100% outside production.

• Match on the path, not the span name. When the sampler runs, the HTTP span is named GET: the framework hasn't matched a route yet. Read url.path (or http.target on older instrumentations) from the attributes. A policy written against span names never fires.
• Children follow their local parent. The sampler is called for every span, and re-rolling each one ships a quarter of a waterfall.
• Decide again at the service boundary. A plain ParentBased sampler trusts the caller's sampled flag. A browser that propagated sampled=0 once switched off every server-side trace for months, and a caller propagating sampled=1 walks past every per-endpoint cap.
• Declare the rate on each span you keep, or every count, throughput and error-rate denominator reads low by the same factor. See Sampling for what knotel reads.

import { SamplingDecision, TraceIdRatioBasedSampler, type Sampler } from "@opentelemetry/sdk-trace-base";
import { trace, type Context, type Attributes } from "@opentelemetry/api";

function rateFor(attrs: Attributes): number {
  if (process.env.ENVIRONMENT !== "production") return 1;
  const path = String(attrs["url.path"] ?? attrs["http.target"] ?? "").split("?")[0];
  if (/^\/health(\/|$)/.test(path)) return 0;
  if (path === "/config") return 0.01;   // 90% of requests, all alike
  return 1;
}

export const sampler: Sampler = {
  shouldSample(ctx: Context, traceId: string, _name: string, _kind, attributes: Attributes) {
    const parent = trace.getSpanContext(ctx);
    // Same process: follow the parent, so a trace is whole or absent.
    if (parent && !parent.isRemote) {
      return { decision: parent.traceFlags & 1 ? SamplingDecision.RECORD_AND_SAMPLED : SamplingDecision.NOT_RECORD };
    }
    // A root, or a caller's context: decide here, whatever the caller did.
    const rate = rateFor(attributes);
    if (rate <= 0) return { decision: SamplingDecision.NOT_RECORD };
    const result = new TraceIdRatioBasedSampler(rate).shouldSample(ctx, traceId);
    return rate < 1 && result.decision === SamplingDecision.RECORD_AND_SAMPLED
      ? { ...result, attributes: { "sampling.probability": rate } }
      : result;
  },
  toString: () => "KnotelEndpointSampler",
};

Keeping everything is a fair choice too, for a service at a few requests a second. One Python service kept ALWAYS_ON on purpose, so no upstream decision can silently turn it off, and trims volume with ingest filters instead.

Queues, crons and idle workers

Work that doesn't arrive over HTTP has no server span to hang off. Give each message or tick its own root, so it shows as one trace with its database calls under it, and record failures on it:

import { trace, SpanKind, SpanStatusCode, type Attributes } from "@opentelemetry/api";

const tracer = trace.getTracer("worker");

export function withRootSpan<T>(name: string, kind: SpanKind, attributes: Attributes, fn: () => Promise<T>) {
  return tracer.startActiveSpan(name, { root: true, kind, attributes }, async (span) => {
    try {
      return await fn();
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

// One trace per message:
subscription.on("message", (msg) =>
  withRootSpan("pubsub.process", SpanKind.CONSUMER,
    { "messaging.system": "gcp_pubsub", "messaging.message.id": msg.id },
    () => handle(msg)));

• Use CONSUMER for messages and INTERNAL for timers, so the service map and request counts treat them right. If the producer put a traceparent in the message's attributes, extract it and add a span link to the producer rather than a parent: one publish can fan out to many consumers.
• A pull loop's own calls — the Pub/Sub pull, the ack, a timer's housekeeping — become root client spans with no request above them. A sampler that returns 0 for a root CLIENT span drops them without touching anything inside a request.
• A worker that's idle sends nothing, which looks the same as a worker that's dead. A memory.heartbeat span every 15 minutes on an unref'd timer tells the two apart and carries a memory reading.

Flush on shutdown, after the drain

Cloud Run sends SIGTERM on every deploy and every scale-in, and Kubernetes on every rollout. The last batch is lost unless something flushes it. Two details decide whether that works:

• A signal listener replaces the default exit. Once you handle SIGTERM, the process no longer exits on it, so exit yourself after the flush — and bound the flush, so a knotel that's unreachable can't hold the shutdown open.
• Flush after requests drain, not when the signal arrives. A framework that drains in-flight requests on SIGTERM (NestJS with shutdown hooks, a Fastify close) is still producing spans. Shut the SDK down at the end of that, in OnApplicationShutdown or after app.close(); shutting it down at the signal drops every span from the drain.

process.once("SIGTERM", async () => {
  await app.close();                                   // drain first
  await Promise.race([sdk.shutdown(), new Promise((r) => setTimeout(r, 2000))]);
  process.exit(0);
});

Serverless platforms that freeze the CPU after the response need more than this; see Cloud Run and Cloud Functions.

Alongside Sentry

Sentry's Node SDK from v8 on is an OpenTelemetry SDK: it registers the global tracer provider, sampler, propagator and context manager. Start a second provider next to it and one of the two creates no spans, depending on which registered first. Pick one owner:

• knotel owns tracing. Keep Sentry for errors only: initialise it with skipOpenTelemetrySetup: true and no tracesSampleRate. Errors are also on knotel's Errors page, grouped, if a failed span records the exception.
• Drop Sentry. A three-line captureException that records the exception on the active span and logs it keeps call sites unchanged.
• Sentry owns tracing. Add knotel's exporter to Sentry's provider as one more span processor, so both receive the same spans.

import { trace, SpanStatusCode } from "@opentelemetry/api";

export function captureException(err: unknown) {
  const span = trace.getActiveSpan();
  const e = err instanceof Error ? err : new Error(String(err));
  span?.recordException(e);
  span?.setStatus({ code: SpanStatusCode.ERROR, message: e.message });
  console.error(e);
}

Python's sentry_sdk keeps its own tracing pipeline, separate from OpenTelemetry's, so the two coexist there — at the cost of sampling twice.

Logs are a separate switch

Turn traces on everywhere and logs on where you'll read them. Logs are the expensive signal: nothing samples them away, each export is a database write, and they're kept for fewer days than spans. Behind a KNOTEL_LOGS=true flag, with the log exporter's batch capped so a killed pod loses little:

new BatchLogRecordProcessor(
  new OTLPLogExporter({ url: config.logs, headers: { "x-knotel-key": config.key } }),
  { maxExportBatchSize: 256 },
);

Where you don't export logs, still stamp trace ids on them (PinoInstrumentation, or Python's LoggingInstrumentor(set_logging_format=False)) so a line in your cloud console leads back to its trace. See Send logs.

Deploys that drop your variables

After load order, the most common way a service stopped reporting was a deploy that quietly removed KNOTEL_*:

Rollout checklist

• KNOTEL_ENDPOINT and KNOTEL_KEY, off unless both are set, trailing slash stripped, one key per environment.
• The header is x-knotel-key. x-flaretrace-key, from before the rename, is refused with 401; old ft_ keys themselves still work.
• The SDK is preloaded, or the first import, and nothing it depends on loads a patched library early.
• A boot line says on or OFF, and why.
• service.name per entry point, deployment.environment.name from ENVIRONMENT, the commit baked in at build.
• Health checks and the exporter's own requests create no spans; fs, dns and net are off.
• Any sampling declares its rate.
• Queue consumers and crons open a root span per unit of work.
• SIGTERM flushes after the drain, with a timeout, then exits.
• After the deploy, the service shows on the project overview within a minute — checked, not assumed.