Operations playbook

This page is for the operator after installation. Once the daemon is up and your policies are in place, what do you actually do with it?

Daily

Two minutes, every morning (or whenever you sit down):

1. Check the dashboard for red badges on tunnels, bridges, or subsystems.
2. Scan active alerts — gwctl alerts active or the Alerts page. Resolve or acknowledge anything that’s no longer real.
3. Confirm last backup ran — Backup page, “Last successful run” timestamp.

That’s it. The daemon does the rest.

Weekly

Ten minutes, once a week:

1. Look at the audit log — gwctl audit or the Audit page filtered by the last week. You’re looking for changes you didn’t expect: new exceptions, new API keys, new federation peers, anyone tampering.
2. Check drift events — gwctl drift (or Reconciliation page). Repeated drift on the same chain means something else on the box is fighting the daemon. Investigate.
3. Run a leak check on the network — gwctl leak-check (or the Privacy → Leak monitor button). Catches resolver misconfigurations before they become incidents.
4. Review tunnel handshake ages — gwctl tunnels. A handshake older than the keepalive interval on more than one tunnel suggests upstream provider issues.

Monthly

Half an hour, once a month:

1. Verify a backup restores. Pull the latest export to a throwaway VM. For the sanitized plaintext export, gwctl backup import <file.json> directly. For an encrypted export, decrypt externally first (no encrypted-import endpoint today) and then gwctl backup import the resulting plaintext. The worst time to discover a corrupted backup is during an actual incident.
2. Review the leak matrix. gwctl leak-matrix. Make sure the posture × tunnel-state grid still matches your intent. Policies you forgot you had can quietly relax the matrix.
3. Review API keys and other secrets — gwctl secrets (the list output is the default; CRUD subcommands are create / get / update / delete). Or Settings → API keys in the UI. Revoke anything you don’t recognise or that an operator left behind. Rotate the long-lived ones.
4. Review federation drift and recent events (if you federate) — GET /api/v1/federation/overview (rolls up drifting peers via peer_drift_summary) and /federation/recent-events (every push that overwrote local state). Drift between peers indicates two operators editing the same kind on different nodes; align their per-peer sync_resources / allowed_receive_resources allowlists.
5. Read the daemon’s release notes for the version you’re running and the one above it. Update if there’s a security fix or a feature you’d benefit from.

Quarterly

Two hours, once a quarter:

1. Capacity check. Compare resident memory and CPU baseline against last quarter (Dashboard → Platform). A trend up means it’s time to think about scaling — see Sizing & performance.
2. Threat model refresh. Has anything changed about your adversaries, providers, or deployment context? If yes, revisit the threat model and check whether the posture still matches.
3. Re-validate alert routing. Send a synthetic alert through every dispatch channel (webhook, Telegram). Real incidents are the wrong time to discover a stale webhook URL.
4. Rotate backup encryption passphrase. Run a new export with the new passphrase; keep one cycle of old-passphrase backups for safety; verify a restore under the new passphrase works; retire the old passphrase.

On-call signals

What should actually wake you up?

Rule (built-in)	Wake you?	Why
leak-detected	Yes, immediately	Active leak; fix before more data exits.
killswitch-engaged	Yes, immediately	A protected policy is in its drop state right now.
direct-fallback	Yes, immediately	A protected device fell back to direct egress. Investigate immediately.
tunnel-down	Page if any fail-closed policy depends on it; notify otherwise	Look at which policies bind to the tunnel.
dns-plane-down	Yes, immediately	Every client’s DNS is broken.
drift-detected (recurring)	Yes	Something else on the box is fighting the daemon. One-off drift is fine; recurring drift is a signal.
cert-expired / cert-expiry-critical	Yes (within hours)	API + UI break when certs expire.
resource-exhaustion	Yes (within hours)	Audit + DNS log + backup are all writing. SQLite WAL can corrupt if disk fills.
tunnel-flap	No (notify)	Repeated cycles often mean a flaky upstream; address during business hours.
dns-bypass	No (notify)	A device tried to talk to 8.8.8.8; not urgent unless repeated.
port-scan	No (notify)	Routine internet noise on a public IP; flag if from inside.
new-device	No (info)	LAN inventory event. Keep for audit.
Daemon stopped responding to /health	Yes, immediately	Whole box is dark.

A good rule: page on guarantees the user expects you to hold. Notify on degradation.

Monitoring stack patterns

The Gateway emits its state via the REST API and the structured journalctl log. Three integration patterns work well:

Webhook → on-call router

Every fired alert hits one webhook URL. Plug it into PagerDuty / Opsgenie / a custom router for paging. Configure once:

curl -X PUT https://gateway.lan:8080/api/v1/settings \
  -H "Authorization: Bearer $API_KEY" \
  -d '{ "webhook_url": "https://oncall.example.invalid/hook/abc123" }'

Route per-severity at the receiver (most on-call routers do this natively). For “page only on critical”, configure your router to drop severity: warning and info.

Telegram bot → operator chat

Single Telegram destination receives the same fires:

curl -X PUT https://gateway.lan:8080/api/v1/settings \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "telegram_bot_token": "REDACTED",
    "telegram_chat_id":   "@my-bot"
  }'

If you want only the notify-worthy alerts in Telegram, disable the critical-only rules you don’t want there at the rule level — and / or split severity at the receiver.

Periodic poll → Prometheus

The daemon exposes plain /api/v1/status, /api/v1/observer, /api/v1/tunnels — scrape every 30 seconds with a small Python script that converts to Prometheus exposition format. Useful for long-term capacity trends.

Common operator mistakes

Things that look right but bite later:

• Putting the LAN in auth.local_net. The bypass only elevates GET/HEAD/OPTIONS (so a hostile LAN device can’t POST /system/wipe blindly), but it still exposes the audit log, DNS query log, full policy graph, device inventory and configuration shape to anyone with a shell on the wider CIDR. Keep it loopback-only unless you trust every device on the LAN to read everything.
• Wide dest_domain: * policies at high priority. They swallow more specific matches; you’ll be confused why a “send X through bridge” policy isn’t firing.
• Fail-open everywhere. Default to fail-closed for anything sensitive; you can soften it later. Leaks during tunnel cycle are easy to miss.
• No backup passphrase rotation. AES-256-GCM is great until your passphrase ends up in a chat log or a stale shell history.
• Skipping the audit log. It’s the one tamper-evident record of who did what; treat it as a primary signal.

Related

• Alerts & monitoring — alert categories, conditions, dispatch.
• Sizing & performance — when capacity numbers say it’s time to scale.
• Troubleshooting — when something is wrong.
• Threat model — the posture the alerts above defend.