Alerts and webhook signatures

Get a signed webhook when runs fail, check its signature on your server, and sign the deliveries you send to a trigger the same way.

What you'll build

A small server that takes shiftlabs alert webhooks, checks each one's Shiftlabs-Signature, and logs the alert. Then a sender that signs its own deliveries to a trigger with the same header, so the trigger starts runs only for requests you signed.

Run the receiver
On an HTTPS address of your own, with the signing secret in its environment.
Point alerts at it
PUT /v1/alerts saves the webhook and returns the signing secret once.
Send a test alert
Then read the delivery log.
Sign your own deliveries
A trigger with shiftlabs signing checks the same header.

Check the signature

Each alert carries Shiftlabs-Signature: t=<unix seconds>,v1=<hex>. v1 is HMAC-SHA256 of <t>.<body>, keyed with the whole secret, whsec_ included. Check it against the raw body, before parsing it, and refuse a t more than 5 minutes off, so a captured request can't be sent again later.

receiver.ts

import { createHmac, timingSafeEqual } from "node:crypto";
import { createServer } from "node:http";

const secret = process.env.SHIFTLABS_ALERT_SECRET ?? "";

function verify(header: string, body: string): boolean {
  const parts = Object.fromEntries(
    header.split(",").map((part) => part.trim().split("=")),
  );
  const t = Number(parts.t);
  if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > 300) {
    return false;
  }
  const expected = createHmac("sha256", secret)
    .update(`${t}.${body}`)
    .digest();
  const given = Buffer.from(parts.v1 ?? "", "hex");
  return given.length === expected.length && timingSafeEqual(given, expected);
}

createServer((req, res) => {
  const chunks: Buffer[] = [];
  req.on("data", (chunk: Buffer) => chunks.push(chunk));
  req.on("end", () => {
    const body = Buffer.concat(chunks).toString("utf8");
    const header = String(req.headers["shiftlabs-signature"] ?? "");
    if (!verify(header, body)) {
      res.writeHead(401).end();
      return;
    }
    const alert = JSON.parse(body);
    console.log(alert.type, alert.test ? "(test)" : "", alert.text);
    res.writeHead(204).end();
  });
}).listen(8080);

import hashlib
import hmac
import json
import os
import time
from http.server import BaseHTTPRequestHandler, HTTPServer

SECRET = os.environ["SHIFTLABS_ALERT_SECRET"].encode()


def verify(header: str, body: bytes) -> bool:
    parts = dict(p.strip().split("=", 1) for p in header.split(",") if "=" in p)
    try:
        t = int(parts["t"])
    except (KeyError, ValueError):
        return False
    if abs(time.time() - t) > 300:
        return False
    signed = f"{t}.".encode() + body
    expected = hmac.new(SECRET, signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts.get("v1", ""))


class Alerts(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("content-length", 0)))
        if not verify(self.headers.get("shiftlabs-signature", ""), body):
            self.send_response(401)
            self.end_headers()
            return
        alert = json.loads(body)
        print(alert["type"], "(test)" if alert["test"] else "", alert["text"])
        self.send_response(204)
        self.end_headers()


HTTPServer(("", 8080), Alerts).serve_forever()

The webhook URL must be https on port 443 at a public hostname, with no user name or password in it, and not on shiftlabs.org. Put the receiver behind your own HTTPS proxy or load balancer.
Answer within 10 seconds. A 5xx, a 429, a timeout or a network error is tried again 1, 5, 15 and 60 minutes later; any other answer is final. Every attempt sends the same body with the same id, so use id to skip an alert you already handled.

Point alerts at it

An admin key saves the alert settings. When you save a webhook and the workspace has no signing secret yet, the answer includes signingSecret; store it as SHIFTLABS_ALERT_SECRET where the receiver runs. GET /v1/alerts never returns it.

shellcurl

# Answers the settings, with "signingSecret": "whsec_…" this once.
curl -X PUT https://api.shiftlabs.org/v1/alerts \
  -H "authorization: Bearer $SHIFTLABS_API_KEY" \
  -H "content-type: application/json" \
  -d '{"emails": [], "webhookUrl": "https://alerts.example.com/shiftlabs",
       "workflows": null, "refusals": true}'

# One test a minute. The receiver logs: runs.failed (test) …
curl -X POST https://api.shiftlabs.org/v1/alerts/test \
  -H "authorization: Bearer $SHIFTLABS_API_KEY"

# The last 20 attempts are in "deliveries", newest first.
curl https://api.shiftlabs.org/v1/alerts \
  -H "authorization: Bearer $SHIFTLABS_API_KEY"

PUT /v1/alerts replaces every setting, so send the emails you want to keep. "workflows": null alerts on every workflow, or name up to 50. "refusals": true adds runs refused at a limit.
Waiting approval requests are sent to the same webhook as approvals.waiting, unless you send "approvals": false. See Alert webhooks for both bodies.
POST /v1/alerts/rotate-secret returns a new secret, and the old one stops working at once. An alert the receiver refuses with 401 before you update it isn't sent again, so update the receiver right after.

Check it worked

The receiver
Logs runs.failed (test) with the alert's text, and answers 204.
The delivery log
GET /v1/alerts lists the attempt in deliveries with status 204 and final true. A 401 there means the receiver's secret isn't the one saved; a retryAt means another attempt is due.

Sign your own deliveries

A trigger created with {"signing": {"scheme": "shiftlabs"}} checks the same header. Its answer returns signingSecret once. A delivery without a valid signature gets 401 {"error": "invalid signature"} and starts nothing.

shellcurl

# Answers 201 {"triggerId", "url", "signingScheme": "shiftlabs",
# "signingSecret"}.
curl -X POST https://api.shiftlabs.org/v1/triggers \
  -H "authorization: Bearer $SHIFTLABS_API_KEY" \
  -H "content-type: application/json" \
  -d '{"workflow": "orders", "signing": {"scheme": "shiftlabs"}}'

send.ts

import { createHmac } from "node:crypto";

const body = JSON.stringify({ orderId: "ord_123" });
const t = Math.floor(Date.now() / 1000);
const v1 = createHmac("sha256", process.env.TRIGGER_SECRET ?? "")
  .update(`${t}.${body}`)
  .digest("hex");

const r = await fetch(process.env.TRIGGER_URL ?? "", {
  method: "POST",
  headers: {
    "content-type": "application/json",
    "shiftlabs-signature": `t=${t},v1=${v1}`,
    // A retry signs again with a new t; this key makes it a duplicate.
    "idempotency-key": "ord_123",
  },
  body,
});
console.log(r.status, await r.json());

import hashlib
import hmac
import json
import os
import time
import urllib.request

body = json.dumps({"orderId": "ord_123"}).encode()
t = int(time.time())
secret = os.environ["TRIGGER_SECRET"].encode()
v1 = hmac.new(secret, f"{t}.".encode() + body, hashlib.sha256).hexdigest()

request = urllib.request.Request(
    os.environ["TRIGGER_URL"],
    data=body,
    headers={
        "content-type": "application/json",
        "shiftlabs-signature": f"t={t},v1={v1}",
        # A retry signs again with a new t; this key makes it a duplicate.
        "idempotency-key": "ord_123",
    },
)
with urllib.request.urlopen(request) as r:
    print(r.status, json.load(r))

The trigger knows a delivery it has seen by its signature, or by its Idempotency-Key. A retry you sign again has a new signature, so send the same Idempotency-Key to make it answer 200 {"duplicate": true, "runId"} instead of starting a second run.
PATCH /v1/triggers/:id with {"signing": {"scheme": "shiftlabs"}} makes a new secret and returns it once; the old one stops working. {"signing": null} stops checking signatures.

Reference

Alert webhooks
Both alert bodies, retries and the delivery log.
Alert emails
Who gets alert emails, and how each member stops them.
Deploy and triggers
Signing schemes and duplicates.
Limits
Alert emails per day on each plan.