Troubleshooting

import { Aside } from ‘@astrojs/starlight/components’;

This is a playbook, not a manual. Each section starts with the symptom you’d actually notice, then walks through what to check, in order, from cheapest to most invasive.

The dataplane in 90 seconds

When something is wrong, the first three commands are almost always:

gwctl status                                    # daemon up? subsystems ok?
gwctl observer                                  # which subsystem is unhappy?
journalctl -u gatewayd --since '5 min ago' -p warn  # the last few warnings

Almost every diagnostic below assumes you’ve run those first.

---

Symptom: a device suddenly has no internet

1. Is the device matched by a fail-closed policy whose tunnel is down?

   gwctl evaluate <device-id-or-mac>
   gwctl tunnels                       # check the named tunnel's handshake age

   If the policy’s tunnel is down and fail_policy: fail-closed, this is working as designed — the kill-switch is doing its job. Either bring the tunnel up or add a fallback policy with lower priority.

2. Is the device’s DHCP lease expired?

   gwctl dhcp leases                   # MAC, IP, hostname, lease expiry
   gwctl devices                       # confirm the device is registered (ID, name, MAC, segment, profile)

   No lease → no L3 → no traffic. Check the DHCP server.

3. Did a recent policy push break routing?

   curl -s https://gateway.lan:8080/api/v1/audit -H "Authorization: Bearer $API_KEY" | jq 'map(select(.resource_type == "routing_policy"))'
   gwctl snapshots                                    # list available
   gwctl rollback <snapshot>                          # back to a known-good state

---

Symptom: the kill-switch won’t disengage even though tunnels are up

The kill-switch is reference-counted per (tunnel, interface) pair. If the counter is stuck above zero, the drop rule stays installed.

1. Inspect the counter:

   gwctl privacy-classes               # which classes reference the tunnel?
   curl -s https://gw.local:8080/api/v1/privacy/kill-switch | jq

2. Force an enforcement re-apply (the trigger is on gwctl enforce; gwctl reconcile only prints status):

   gwctl enforce reconcile

3. If the counter is wrong, the daemon will rebuild it from policy state on restart:

   sudo systemctl restart gatewayd

   Restart drops conntrack state for any flow tagged by the splitting engine. Expect a brief disruption.

---

Symptom: traffic isn’t using the tunnel I expect

1. Run the evaluator — what does the daemon think the policy is?

   gwctl evaluate <device-id-or-mac>

2. Compare against compiled state:

   gwctl render                       # full compiled nftables / ip-rule output

3. Look for a higher-priority policy that’s stealing the match. Open the Routing policies page and sort by priority, or query the API:

   curl -s https://gw.lan:8080/api/v1/routing/policies \
     -H "Authorization: Bearer $API_KEY" \
     | jq 'sort_by(-.priority)'

   First match wins; a wildcard dest_domain: * policy at priority 100 swallows everything below it.

4. Check exceptions: a TTL’d override may be active.

   gwctl exceptions

---

Symptom: DNS queries leak to the ISP

1. Confirm the device is going through the gateway resolver. Open the DNS log page in the UI, or query the cache via the REST API and filter by device — there’s no gwctl wrapper for the cache:

   curl -s https://gateway.lan:8080/api/v1/dns/cache \
     -H "Authorization: Bearer $API_KEY" \
     | jq 'map(select(.client_ip=="10.0.10.50"))'

   If you see nothing for that device, it isn’t using the resolver at all.

2. Is force_dns enabled for the device’s privacy class? Without it, a hard-coded 8.8.8.8 bypasses the integrated resolver.

3. Is DoH being blocked? Check Settings → Privacy → block_doh. A browser using DoH won’t show up in the DNS log at all.

4. Run the leak checker — there’s no per-device flag; the probe runs at network scope and the report tells you which test failed and where:

   gwctl leak-check

   Result is a LeakReport JSON with one entry per probe (dns_leak, dns_bypass, dns_forcing, dns_upstream_route, dnssec, doh_blocked, ipv6_leak, ipv4_fallback, webrtc_stun) — see Privacy → leak checker.

---

Symptom: bridge is unhealthy

1. Reachability: there’s no gwctl wrapper for bridges; use the REST API:

   curl https://gateway.lan:8080/api/v1/bridges \
     -H "Authorization: Bearer $API_KEY"           # list with health column
   curl -X POST https://gateway.lan:8080/api/v1/bridges/<bridge-id>/test \
     -H "Authorization: Bearer $API_KEY"           # one-shot probe

2. Check the bridge-node logs on the VPS:

   ssh vps "journalctl -u bridge-node -n 200"

   Common: certificate expired, PSK mismatch, decoy directory not readable.

3. Decoy SNI mismatch: if the SNI on the wire doesn’t match the bridge-node’s TLS cert, the censor’s gear sees a TLS error and your client too. Confirm the bridge’s decoy_domain is a real hostname on the bridge-node’s cert.

---

Symptom: throughput is much lower than the link

1. Privacy level at fault?

   gwctl splitting                     # shows per-policy level + slot map

   Level 3+ adds HTB shaping. The shape rate is the ceiling.

2. MTU mismatch. Check the tunnel’s MTU on the Tunnels page or via gwctl tunnels. If MTU is 1500 but the tunnel’s effective MTU is 1380, you’ll see fragmentation and retransmits. Set the tunnel’s MTU explicitly.

3. CPU-bound WireGuard: the Pi 4 caps at ~300 Mbit per core for WireGuard. Splitting across multiple tunnels helps spread load.

4. Cover traffic competing with real traffic? Cover lives on its own HTB class so this shouldn’t happen — but if splitting.cover_traffic_max_pps was raised manually, it can.

---

Symptom: the reconciler keeps re-applying state

1. Something else is editing nftables / ip rule:

   gwctl drift

   Each drift entry includes the diff. If you see the same rule being added by drift over and over, find what’s removing it (typical suspects: Docker’s iptables, a stray ufw enable, a sibling daemon).

2. Quiet mode: raise daemon.reconcile_interval_sec to reduce churn while you investigate. Don’t lower the kill-switch’s coverage to make the alerts stop.

---

Symptom: federation isn’t syncing

1. Tunnel between peers up?

   gwctl federation peers

   If the peer’s WireGuard handshake is stale, fix the tunnel first.

2. Per-peer sync allowlists set the way you intend?

   gwctl federation-roles                       # labelling taxonomy (descriptive only)
   curl https://gateway.lan:8080/api/v1/nodes \
     -H "Authorization: Bearer $API_KEY"        # see sync_policy + sync_resources + allowed_receive_resources per peer

   If the receiving peer’s allowed_receive_resources doesn’t include the kind being pushed, the push is silently dropped. Update both sides via PUT /api/v1/nodes/{id} — see Federation.

3. Recent events + drift: open the Federation page (or GET /api/v1/federation/recent-events and /federation/overview). Every overwrite from an incoming push is logged, and drifting peers are surfaced via the peer_drift_summary rollup. Conflicts are resolved last-writer-wins; the overwritten row appears in the event log for inspection.

---

Capture targeted traffic

When a symptom doesn’t fit a chapter above, capture the raw packets. The daemon wraps tcpdump with policy context, capture presets and per-device filters — no shell needed.

Quick captures from the CLI

gwctl capture start --iface wg-eu-01 --filter "port 53" --count 200    # 200 DNS packets
gwctl capture list                                                       # active + recent
gwctl capture get <id>                                                   # JSON with parsed summaries
gwctl capture stop <id>                                                  # end early

Presets

The daemon ships fourteen named BPF filters so you don’t have to remember tcp[tcpflags] & ... for common scenarios:

Preset key	Filter	Use for
dns	port 53	DNS queries + answers
http	tcp port 80 or tcp port 443	Web egress
icmp	icmp or icmp6	Ping, traceroute, unreachables
dhcp	port 67 or port 68	DHCP discovery + leases
arp	arp	L2 address resolution
tcp_syn	tcp-syn only	New connection attempts
tcp_rst	tcp-rst / tcp-fin	Failures and teardowns
udp_no_dns	udp and not port 53	UDP minus DNS noise
wireguard	udp port 51820	WireGuard handshakes
tailscale	tcp port 443 or udp port 3478 or udp port 41641	Tailscale / DERP
ntp	udp port 123	Time sync
ssh	tcp port 22	SSH
broadcast	ether broadcast or ether multicast	Noisy LAN frames
large_pkt	greater 1400	MTU / fragmentation

Reference a preset via the REST API:

curl -X POST https://gateway.lan:8080/api/v1/diagnostics/capture \
  -H "Authorization: Bearer $API_KEY" \
  -d '{"interface":"wg-eu-01","preset":"dns","device_ip":"10.0.10.50","count":200,"duration_sec":60}'

device_mac or device_ip narrows the capture to one LAN client — useful when one device is misbehaving in a crowd.

Limits

The capture engine is intentionally bounded:

• max 3 concurrent captures
• max 20 retained sessions
• max 50 000 packets per session
• max 3600 s per session

Hit a limit, the API rejects with 429 or 400; stop an old session first.

Stream live

curl -N https://gateway.lan:8080/api/v1/diagnostics/capture/<id>/stream \
  -H "Authorization: Bearer $API_KEY"

NDJSON, one parsed packet per line. The Capture page in the UI consumes the same stream.

---

When all else fails

Capture a manual diagnostic bundle:

mkdir -p /tmp/gw-bundle && cd /tmp/gw-bundle
journalctl -u gatewayd --since '24 hours ago' > logs.txt
gwctl status > status.txt
gwctl observer > observer.txt
gwctl tunnels > tunnels.txt
curl -s https://gateway.lan:8080/api/v1/bridges -H "Authorization: Bearer $API_KEY" > bridges.txt
curl -s https://gateway.lan:8080/api/v1/routing/policies -H "Authorization: Bearer $API_KEY" > policies.txt
gwctl drift > drift.txt
gwctl audit > audit.txt
sudo nft list table inet gateway > nftables.txt
ip rule list > ip-rule.txt
ip -all route show table all > ip-route.txt
cd .. && tar czf gw-bundle.tar.gz gw-bundle/

Inspect before sharing — the bundle includes device hostnames, tunnel names and policy structure (no keys or backup contents).

Related

• Architecture — how the dataplane fits together; useful when reading diagnostics.
• Concepts — vocabulary.
• FAQ — the questions that aren’t symptoms.