Alerts

An alert is a threshold on a number the dashboard already shows - error rate, p95, requests, or how many log lines matched - checked on a schedule and posted to a webhook (Slack or your own) when it crosses. There's no second query language: the filters are the same ones the Traces page puts in its URL.

Create an alert

Open a project's Alerts page and choose New alert. The editor is a page of its own that walks through what to watch, when it counts as broken, how patient to be and which webhook to tell, with a summary of the finished rule beside it. Next to the threshold it shows what the rule measures right now, from the same query the check runs, so the line is drawn against the real number. Each alert watches one of two things:

Choose All projects and one rule covers every project, with its own state in each: one broken project is one incident to announce and to recover, not a number averaged with nine healthy ones until it stops crossing the line.

When it fires

• No data is not a breach. A window with nothing in it is skipped, not read as zero, so a "requests below 10" alert doesn't fire on an idle weekend. To catch a service going silent, watch a busier scope, or rely on Home's Went quiet signal.
• A resolve is sent once, when a firing alert comes back under the line. A breach that never got past pending was never announced, so it isn't resolved either.
• Muting stops sending, not checking. Mute for an hour across a deploy and an alert that broke and recovered inside it comes out quiet, rather than announcing a fire that's already over.

Webhooks

Alerts post to webhooks. A webhook is set up once, under Webhooks or right in the alert editor, and shared by every project. Each has a Send test that goes through the real delivery path, and the page remembers whether the last test worked. Webhooks are the only way alerts are sent for now; there is no email.

Slack

Create an incoming webhook in Slack and paste its URL as a webhook; knotel recognises it and picks the Slack format. Messages carry the condition, the measured value, the threshold and a button back to the project.

Your own receiver

The JSON format posts:

{
  "alert": "Checkout error rate",
  "state": "fired",            // or "resolved"
  "project": "shop",
  "projectId": "p_…",
  "condition": "error rate above 2% over 15 min",
  "value": 4.7,                // null in a test send
  "threshold": 2,
  "at": "2026-09-22T03:14:00.000Z",
  "link": "https://trace.example.com/p/p_…"
}

Generate a signing secret and each request carries x-knotel-signature: sha256=<hex>, the HMAC-SHA256 of the raw body. Check it before trusting the payload:

import { createHmac, timingSafeEqual } from "node:crypto";

function verified(rawBody: string, header: string | undefined, secret: string) {
  const expected = "sha256=" + createHmac("sha256", secret).update(rawBody).digest("hex");
  return !!header && header.length === expected.length &&
    timingSafeEqual(Buffer.from(header), Buffer.from(expected));
}

Extra headers, such as an Authorization your receiver wants, can be added to the destination. A request times out after 10 seconds; a 5xx or 429 is retried once, and any other error is not, since it won't fix itself.

What alerts cost

Checking is the cost, in ClickHouse CPU; sending is rounding error. Alerts due in the same minute that share a scope share one query, and an all-projects alert is one query grouped by project, not one per project. Each check reads only its look-back window, so a busy instance with many alerts still spends little time on them. Nothing is billed per query or per message.

Alerts worth starting with