Serverless and Kubernetes

An OpenTelemetry SDK assumes a process that starts, runs and stops on its own terms. Serverless platforms and Kubernetes each break a different one of those assumptions, and each breaks it quietly: the service keeps serving, the boot line says tracing is enabled, and spans go missing anyway. This page is the four primitives that decide whether that happens, and what each platform does to them.

The four primitives

Cloud Run and Cloud Functions

Cloud Functions gen2 is Cloud Run underneath, so these are one platform with two front doors, and they share the single most expensive gotcha in this page.

Pick one of three fixes. Flushing before you respond is the one that needs no platform change:

const provider = new NodeTracerProvider({
  resource,
  spanProcessors: [new BatchSpanProcessor(exporter)],
});
provider.register();

// Fastify/Express: last thing before the handler resolves.
app.addHook("onResponse", async () => {
  await provider.forceFlush();
});

• Turn on CPU always allocated and keep the batch processor. Costs more per instance-hour; the SDK then behaves like a normal server.
• forceFlush() before responding (above). One extra round trip on the tail of each request — pair it with sampling so it is not every request.
• SimpleSpanProcessor, which exports each span as it ends. Simple and slow: a POST per span, in the request's own path. Fine for a low-traffic function, wrong for anything busy.

Identity and environment

Cloud Run sets K_SERVICE, K_REVISION and K_CONFIGURATION in the environment. The revision is the useful one: it changes on every deploy, so it answers "did that release do this?" the way service.version does elsewhere.

resourceFromAttributes({
  "service.name": process.env.K_SERVICE ?? "my-function",
  "cloud.platform": "gcp_cloud_run",
  "cloud.region": process.env.FUNCTION_REGION ?? "",
  // Changes every deploy — group by this to compare releases.
  "service.version": process.env.K_REVISION ?? "",
})

Continuation

Google's load balancer sends X-Cloud-Trace-Context, not W3C traceparent. If the caller is another one of your services the W3C header is already there and nothing is needed; if the trace starts at the load balancer, add the GCP propagator so the hop joins instead of starting fresh.

AWS Lambda

Lambda freezes the execution environment the instant the handler returns, and may never thaw it again. Same failure as Cloud Run, sharper edges: flush inside the handler, in a finally, or lose whatever the batch was holding.

export const handler = async (event, context) => {
  try {
    return await realHandler(event, context);
  } finally {
    // Inside the handler, not after: once this resolves the environment freezes.
    await provider.forceFlush();
  }
};

The compensation is that Lambda gives you the cleanest version of everything else. One invocation per environment means concurrency is 1, so a process-wide measure really is this request's — the per-request CPU numbers in CPU and memory are exact here, not contaminated the way they are on a threaded or multi-tenant pod.

// Module scope runs once per environment, so this is true on the first
// invocation of a new one and false for every reuse.
let cold = true;

export const handler = async (event, context) => {
  span.setAttributes({
    "faas.coldstart": cold,
    "faas.invocation_id": context.awsRequestId,
    "faas.name": process.env.AWS_LAMBDA_FUNCTION_NAME,
    // Identifies the environment; the log stream is per-instance.
    "faas.instance": process.env.AWS_LAMBDA_LOG_STREAM_NAME,
  });
  cold = false;
  // ...
};

faas.coldstart is worth setting even if you set nothing else. Cold starts are usually the entire slow tail, and with the attribute on the span, outlier attribution finds that on its own rather than you suspecting it.

Kubernetes

Kubernetes keeps your process alive and running, so the flush problem softens into a shutdown problem — and the identity problem gets much worse, because now several replicas answer the same traffic.

Flush on SIGTERM

Every rolling deploy kills every pod. Without a shutdown flush you lose the last batch on each one, which is precisely the spans of whatever was in flight when the update arrived — the ones worth having.

process.on("SIGTERM", async () => {
  await app.close();        // stop accepting, drain in flight
  await provider.shutdown(); // flush what the batch processor still holds
  process.exit(0);
});

Identity: which pod answered

This is the one that matters. With several replicas, a single leaking pod and a fleet-wide drift produce an identical average — the numbers only become actionable once each span says which pod it came from.

The pod name needs nothing from your chart: Kubernetes sets the container hostname to it, and KUBERNETES_SERVICE_HOST is injected into every pod, so it doubles as a reliable "am I on a cluster?" test. Namespace and node have no such shortcut and come from the downward API.

env:
  - name: K8S_NAMESPACE
    valueFrom:
      fieldRef: { fieldPath: metadata.namespace }
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef: { fieldPath: spec.nodeName }

import { hostname } from "node:os";

const onCluster = !!process.env.KUBERNETES_SERVICE_HOST;

resourceFromAttributes({
  "host.name": hostname(),
  // Only claimed on a cluster: a laptop's hostname is not a pod name.
  ...(onCluster ? { "k8s.pod.name": hostname() } : {}),
  ...(process.env.K8S_NAMESPACE
    ? { "k8s.namespace.name": process.env.K8S_NAMESPACE }
    : {}),
  ...(process.env.K8S_NODE_NAME
    ? { "k8s.node.name": process.env.K8S_NODE_NAME }
    : {}),
})

Concurrency changes what CPU means

A pod serves many requests at once. process.cpuUsage() is process-wide, so a delta measured across one request counts every other request in flight beside it: on a busy pod it stops measuring "what this request cost" and starts measuring "how loaded was the pod". Still useful ranked across thousands of spans; near-meaningless on one.

For a Node service the honest saturation number is event loop utilization, which never claimed to be per-request:

import { performance } from "node:perf_hooks";

const before = performance.eventLoopUtilization();
// ... handle the request ...
const elu = performance.eventLoopUtilization(before);
span.setAttributes({
  "nodejs.eventloop.utilization": Math.round(elu.utilization * 1000) / 1000,
});

Direct, or through a Collector

Every example here posts OTLP straight to https://YOUR-INSTANCE/v1/traces, which is one less moving part and works everywhere. A Collector earns its place on Kubernetes specifically: run it as a DaemonSet, have pods send to the node-local instance, and the batching, retries and the k8sattributes processor (which fills in pod, namespace and node labels for you) stop being every service's problem. On Lambda and Cloud Run, where the process is frozen between requests, a sidecar collector usually just moves the flush problem rather than solving it.

Checklist