Skip to content
The Gateway

Alerts & monitoring

The Gateway watches itself. Three subsystems combine to surface the state of the box:

  • observe — checks subsystem health and detects configuration drift.
  • alert — fires built-in rule IDs when the daemon detects a condition; dispatches via webhook + Telegram.
  • anomalies — heuristic detection of unusual traffic patterns.

Alert rules

Section titled “Alert rules”

There is no user-configurable condition DSL. The daemon ships with 17 built-in rules; you can enable / disable each, but you can’t define new ones or change their trigger logic. The Rule struct is just {id, name, category, severity, enabled}.

Rule ID Category Default severity Fires when
tunnel-down tunnel critical A tunnel’s handshake is older than its keepalive + grace
tunnel-flap tunnel warning A tunnel cycles up/down more than N times in a window
killswitch-engaged privacy critical The kill-switch fired for any policy
dns-bypass dns warning A client tried to talk to an external resolver directly
dns-plane-down dns critical The integrated resolver is unreachable
direct-fallback privacy critical A protected device fell back to direct egress
component-failed system warning A subsystem reported failed health status
drift-detected config warning The reconciler had to re-apply state
auth-failure security warning Failed login attempt against the API or UI
resource-exhaustion system critical Disk / memory / CPU saturation
cert-expiry-critical security critical A TLS cert is within the critical-expiry window
cert-expired security critical A TLS cert has expired
backup_failed system warning A scheduled backup failed locally or its remote push errored
port-scan security warning Port-scan signature seen against the daemon or a tunnel exit
new-device network info A device the gateway has never seen joined the LAN
leak-detected privacy critical The leak monitor returned any failing test
wan-stall network warning The WAN interface stopped passing traffic

Toggle a rule with:

Terminal window
curl -X PUT https://gateway.lan:8080/api/v1/alerts/rules/tunnel-flap \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

That’s the entire mutation surface for rules. There is no PUT for severity, name, category or condition — those are fixed at compile time.

Dispatch

Section titled “Dispatch”

Both channels are configured once, globally, in kv_settings. Every rule fire goes to both (if both are configured); per-rule routing isn’t supported.

Channel kv_settings keys What it does
Webhook webhook_url HTTP POST with a JSON payload of the fired alert
Telegram telegram_bot_token, telegram_chat_id Sends a formatted message to the chat

Set them via Settings → Notifications in the UI or PUT /api/v1/alerts/webhook (which accepts {webhook_url, telegram_bot_token, telegram_chat_id} as plain JSON keys). Use POST /api/v1/alerts/webhook/test to send a synthetic test through the configured channel without waiting for a real fire.

Categories

Section titled “Categories”

For grouping in the UI: tunnel, dns, privacy, config, system, security, network. The category of each rule is fixed in the daemon.

Alert lifecycle

Section titled “Alert lifecycle”

Alerts persist with:

  • fired_at / resolved_at
  • severity
  • source — what triggered it (tunnel id, device MAC, etc.)
  • message, details

Resolution is manual (POST /api/v1/alerts/{id}/resolve) or acknowledged (POST /api/v1/alerts/{id}/acknowledge). Each Manager retains the last 500 alerts in memory.

Observer

Section titled “Observer”

The observer page rolls up subsystem health into one view:

$ gwctl observer


Subsystem    Health   Last check    Note
─────────────────────────────────────────────────────────────
dns          ok       2s ago        cache hit rate 87%
transport    ok       3s ago        2 tunnels up, 0 stale
firewall     ok       5s ago        nftables loaded, no drift
mesh         degraded 1s ago        coordinator RTT 480ms (warn)
storage      ok       11s ago       free 4.2GB of 8GB

Each subsystem exports a small set of metrics and a status enum (ok / degraded / failed). When a subsystem flips to failed, the component-failed alert rule fires.

Drift detection

Section titled “Drift detection”

On every reconciler tick (default 60 s) the daemon computes the desired state from policy / settings and diffs it against the kernel’s actual state (nftables tables, ip rule, routing tables, redsocks processes). Differences are:

  • Re-applied automatically (the kernel is forced to match desired state).
  • Logged to the drift event log with the diff.
  • Optionally alerted on (the built-in drift-detected rule) so a recurring drift gets attention.

This makes the daemon self-healing: a sibling script that flushed an iptables chain, or a partial restart that left orphaned rules, gets corrected on the next reconciler tick.

Anomaly detection

Section titled “Anomaly detection”

The observe subsystem runs four lightweight heuristics for things that don’t fit the rule-based model cleanly. Each emits an Anomaly event surfaced on the Anomalies page and via GET /api/v1/anomalies.

Type Severity Trigger
dns_spike warning A client’s DNS query rate exceeds max(avg + 3σ, avg × 3) against its 30-sample rolling baseline.
traffic_spike warning An interface’s egress bytes exceed 5 × avg over the 30-sample window.
new_device info A MAC address is seen for the first time. The known-devices set is seeded from the device table at boot, so existing devices don’t all alert after a restart.
port_scan critical A single source IP hits ≥ 15 distinct destination ports within the sampling window.

Two implementation choices worth knowing about as an operator:

  • Anomalous samples don’t teach the baseline. A sustained attack that ramped up the DNS rate could otherwise gradually raise the rolling average until the attack rate became “normal” and the detector silently stopped firing. The detector explicitly discards anomalous samples from its baseline update — security-correct default.
  • Warm-up window of 5 samples. During the first 5 samples on a fresh client/interface, no anomaly fires regardless of magnitude; the detector needs a baseline to compare against. Daemon restarts reset this.

Read baseline state for diagnostics via GET /api/v1/anomalies/stats — returns the current rolling mean + stddev per client and per interface, plus the total event count.

Bluetooth IDS

Section titled “Bluetooth IDS”

If the box has a Bluetooth adapter and you’ve enabled the radio (POST /api/v1/bluetooth/enable), a separate heuristic engine watches bluetoothctl scan results for wireless-layer anomalies. It’s distinct from the network-layer anomaly detector above — different inputs (scanned BT devices vs IP traffic), different lifecycle.

Type Severity Trigger
bt_new_device info A MAC not seen during the 30-minute learning window appears in a later scan.
bt_name_change warning A known device’s advertised name changes between scans. Common signal for spoofing attempts.
bt_rssi_spike warning RSSI jumps by more than +20 dBm between consecutive samples for a known device — sudden proximity.
bt_rssi_drop info RSSI falls by more than -30 dBm — device moved away, or jamming.
bt_mac_cycling critical Same OUI prefix produces ≥ 10 new MACs in 5 minutes — BLE random-address rotation by a single hardware device, often surveillance gear.
bt_pair_brute_force critical ≥ 5 pair-attempts in 2 minutes against the box’s own adapter.

Operator behaviour worth knowing:

  • 30-minute learning phase. From daemon start, every device seen feeds the baseline; no bt_new_device / bt_name_change events fire. After 30 minutes the detectors arm. Restart resets the clock.
  • Alert cooldown of 5 minutes per (type, MAC). A single noisy device won’t flood the event log.
  • Ring buffer of 100 events. Read via GET /api/v1/bluetooth/ids, which returns {enabled, learning, baseline_count, events}.
  • The engine runs only if the BT manager is enabled. On boxes without a Bluetooth adapter (or with it disabled in settings) the IDS reports {"enabled": false, "events": []} and consumes no CPU.

The threshold defaults are tuned for “an apartment with consumer devices nearby”. In dense environments (offices, conference centres) you’ll see legitimate bt_mac_cycling events from phones using BLE privacy randomisation — that’s a known false-positive class; treat the OUI as the unit of judgement, not the count alone.

Live logs and audit

Section titled “Live logs and audit”
  • Live logs — journalctl -u gatewayd -f streamed into the UI with level filtering and search.
  • Audit — every config-changing API call is logged with operator, timestamp, before, after, source_ip. Retention is log.retention_days in the bootstrap config (default 90 days).

API surface

Section titled “API surface”
Method Path Notes
GET /api/v1/alerts All alerts (active + resolved)
GET /api/v1/alerts/active Only active
POST /api/v1/alerts/{id}/acknowledge Mark acknowledged
POST /api/v1/alerts/{id}/resolve Mark resolved
GET /api/v1/alerts/rules List built-in rules + enabled state
PUT /api/v1/alerts/rules/{id} Toggle a rule: body {enabled: bool}
GET/PUT /api/v1/alerts/webhook Webhook configuration
POST /api/v1/alerts/webhook/test Send a synthetic test webhook
GET /api/v1/observer Subsystem health snapshot
GET /api/v1/drift Recent drift events
GET /api/v1/anomalies Recent anomalies with time-series
GET /api/v1/anomalies/stats Anomaly counters
GET /api/v1/bluetooth/ids Bluetooth IDS events + learning state
GET /api/v1/audit Audit log

Next steps

Section titled “Next steps”

After dispatch is configured:

  1. Send a synthetic test through every channel. POST /api/v1/alerts/webhook/test proves the URL works now — not when a real leak-detected fires. Repeat quarterly.
  2. Decide the page-vs-notify split before you need it. See the on-call signals table for which built-in rules belong on which tier.
  3. Disable rules you don’t want. Each rule has a {enabled: bool} toggle via PUT /api/v1/alerts/rules/{id} — don’t leave noisy rules on just because they’re built-in.
  4. Wire the anomaly and BT-IDS feeds into the same on-call channel if they’re meaningful for your deployment, or suppress them at the receiver if not.
Section titled “Related”