Diagnostic procedure
Always go from top to bottom - so you won’t restart the service when the network actually crashed: Network → Service → Dependency → Data → Recovery- Network Whether the node/gateway is available over the network (L1: reachability, DNS, ports).
- Service - whether the server part is running (L2, requires an agent).
- Dependencies - WARP, external proxies, CDNs, conditional connections and transit nodes.
- Data - what the diagnostics, incident tapes and log show: what changed before the failure.
- Recovery – just now choose and confirm the action.
Runbook: node in critical
Symptom
critical on the dashboard, the reason is in the card: entry_node_down/transit_node_down, remnanode_down/xray_down, disconnected node in Remnawave or offline agent.Diagnostics
GET .../nodes/{id}/probes). In order: net — has passed the reachability (L1); service — what says L2 (needs an online agent); Data — collect logs through the action view_remnanode_logs/view_xray_logs (read only) and see the journal: did not coincide with the recent change.Proposed action
drain_node (remove the node from the pool), recheck (if the agent is online), and restart the service with runtime (restart_remnanode or restart_xray). If the agent is offline, the restart buttons will not be available with a prompt-cause.Confirmation
POST .../actions/{id}/confirm) If users need to get off the problem node immediately, first drain_node, then deal with the service.Verification of recovery
POST .../nodes/{id}/check; make sure reachability and service are back in ok, and consolidated node status is healthy.When escalate
Runbook: Flow in Critical
Symptom
criticalPossible quorum causes: quorum of input nodes is lost (<50% healthy or 0 healthy), inaccessible gateway, persistent failure of the through test on the part of the node, drift configuration (config_drift).Diagnostics
GET .../connection-units/{id}/diagnostics) - fresh samples, associated log entries and metadata nodes. network (Are gateways and input nodes achievable?) service (Nodes respond to L2?) dependency (transit/routes/WARP/proxy) data (What has changed: configuration application, update, policy)Proposed action
config_drift system will offer reapply_config at the schema level, and to critical - additionally rollback_unit_version. For specific nodes will be offers from runbook nodes (drain/restart/recheck).Confirmation
Verification of recovery
POST .../connection-units/{id}/check (one through passage). The circuit should return to healthy, and a series of unsuccessful checks should reset.When escalate
Runbook: Gateway unavailable
Symptom
critical with the cause of the gateway inaccessibility (gateway_unreachable). As long as the gateway is not available, the input nodes are not important - the path for the user is severed.Diagnostics
Proposed action
recheck). If the problem is networked and outside the node, the recovery is done on the infrastructure or provider side.Confirmation
Verification of recovery
POST .../connection-units/{id}/check: Gateway reachability - ok, circuit - healthy.When escalate
Runbook: Agent offline
Symptom
offline; in the tape - incident by agent, in Telegram - category agent_offline. The agent is marked offline automatically after 90 seconds without heartbeat.Diagnostics
Proposed action
recheck is available and, if necessary, to remove traffic, drain_node.Confirmation
recheck. Recovery of the agent as a service is performed on the node host (see Node Life Cycle).Verification of recovery
offline. Confirm recheck and make sure the node is healthy.When escalate
critical - a maximum of degraded. The real inaccessibility of the node will independently show its reachability tests.Runbook: the degradation of the region (external samples)
Symptom
external_regional_failure.Diagnostics
external_probe_unavailable is not a failure.Proposed action
Confirmation
Verification of recovery
When escalate
Verification
After recovery, related action sentences are removed automatically, and the tape entry goes toauto_resolved.
Rollback
This page describes a process where there is nothing to roll back. The recoils of specific recovery actions are described where these actions are performed:- node withdrawal from the pool (
drain_node) is canceled by actionundrain_node- see Restoration; - re-application of the configuration and rollback of the schema version on the pages of Flow;
- Node suppression is removed on the page Node suppression.
Typical errors
Relaunched the service and the reason was online.
Relaunched the service and the reason was online.
I was waiting for reapply_config/rollback to fix the scheme.
I was waiting for reapply_config/rollback to fix the scheme.
Running recheck, and it doesn't escalate to critical
Running recheck, and it doesn't escalate to critical
