Skip to main content
This page is a set of pre-made scripts (runbooks) for typical critical allerts. Each is arranged in the same way: symptom → diagnosis → proposed action → confirmation → recovery check → when to escalate. Keep the same order of diagnosis and do not fix the symptom at random.
For whom: panel administrator, SRE, duty engineer. Required rights: Administrator session.
This page describes manual operator scripts. Flow Guard allows you to confirm a specific launch or allow automatic plans for Flow in advance. Automation is limited to saved policy, fresh checks and limits.

Diagnostic procedure

Always go from top to bottom - so you won’t restart the service when the network actually crashed: Network → Service → Dependency → Data → Recovery
  1. Network Whether the node/gateway is available over the network (L1: reachability, DNS, ports).
  2. Service - whether the server part is running (L2, requires an agent).
  3. Dependencies - WARP, external proxies, CDNs, conditional connections and transit nodes.
  4. Data - what the diagnostics, incident tapes and log show: what changed before the failure.
  5. Recovery – just now choose and confirm the action.

Runbook: node in critical

1

Symptom

The node is shown as critical on the dashboard, the reason is in the card: entry_node_down/transit_node_down, remnanode_down/xray_down, disconnected node in Remnawave or offline agent.
2

Diagnostics

Open the page of the node and its sample history (GET .../nodes/{id}/probes). In order: net — has passed the reachability (L1); service — what says L2 (needs an online agent); Data — collect logs through the action view_remnanode_logs/view_xray_logs (read only) and see the journal: did not coincide with the recent change.
3

Proposed action

The system will offer a suitable reason: drain_node (remove the node from the pool), recheck (if the agent is online), and restart the service with runtime (restart_remnanode or restart_xray). If the agent is offline, the restart buttons will not be available with a prompt-cause.
4

Confirmation

Confirm the action (POST .../actions/{id}/confirm) If users need to get off the problem node immediately, first drain_node, then deal with the service.
5

Verification of recovery

Run POST .../nodes/{id}/check; make sure reachability and service are back in ok, and consolidated node status is healthy.
6

When escalate

If, after restarting the service, the node falls again, or the agent does not communicate, collect a support bundle and escalate by Response to incidents.

Runbook: Flow in Critical

1

Symptom

The chart is shown. criticalPossible quorum causes: quorum of input nodes is lost (<50% healthy or 0 healthy), inaccessible gateway, persistent failure of the through test on the part of the node, drift configuration (config_drift).
2

Diagnostics

Open the diagram card and its diagnostics (GET .../connection-units/{id}/diagnostics) - fresh samples, associated log entries and metadata nodes. network (Are gateways and input nodes achievable?) service (Nodes respond to L2?) dependency (transit/routes/WARP/proxy) data (What has changed: configuration application, update, policy)
3

Proposed action

To the caus config_drift system will offer reapply_config at the schema level, and to critical - additionally rollback_unit_version. For specific nodes will be offers from runbook nodes (drain/restart/recheck).
4

Confirmation

Re-application and rollback are available through Flow’s regular operations and Flow Guard plans. In the Version tab, the rollback uploads the picture to a draft and requires re-application; the recovery plan can perform the application itself. Before starting, check the steps shown and the impact on the draft. See Application and versions of.
5

Verification of recovery

Run POST .../connection-units/{id}/check (one through passage). The circuit should return to healthy, and a series of unsuccessful checks should reset.
6

When escalate

If the quorum is not restored after repairing the nodes, or the drift of the configuration is repeated after repeated use, escalate with the application of diagnostics and journal.

Runbook: Gateway unavailable

1

Symptom

Circuit in critical with the cause of the gateway inaccessibility (gateway_unreachable). As long as the gateway is not available, the input nodes are not important - the path for the user is severed.
2

Diagnostics

Start with network: whether the basic reachability of the gateway (L1 - DNS, TCP to the desired ports). Then service on the gateway node (L2), then dependency (whether the path is blocked by a firewall / provider).
3

Proposed action

If the gateway node itself is unavailable, use runbook nodes (Run the service with runtime, recheck). If the problem is networked and outside the node, the recovery is done on the infrastructure or provider side.
4

Confirmation

Confirm the service restart only if L2 shows that the service has fallen. If the network is external, eliminate it first, action in the panel will not help here.
5

Verification of recovery

POST .../connection-units/{id}/check: Gateway reachability - ok, circuit - healthy.
6

When escalate

If the gateway is available over the network, but the circuitry remains critical, it’s not a gateway anymore, you can look at the runbook circuitry. If the inaccessibility is networked and out of your control, escalate to the provider/infrastructure.

Runbook: Agent offline

1

Symptom

The agent of the node is shown offline; in the tape - incident by agent, in Telegram - category agent_offline. The agent is marked offline automatically after 90 seconds without heartbeat.
2

Diagnostics

Key question: is the node alive, but the agent is silent - or did the node really fall? network: Achievability tests (L1) come from the panel side and dependency If L1 is OK and L2 isn’t available, the problem is probably the agent itself, not the node.
3

Proposed action

While the agent is offline, actions requiring the agent (restarts, log collection, WARP installation), is not available — gray buttons with a cause. recheck is available and, if necessary, to remove traffic, drain_node.
4

Confirmation

If the L1 sample is successful, it is usually enough to wait for the agent to return and perform recheck. Recovery of the agent as a service is performed on the node host (see Node Life Cycle).
5

Verification of recovery

Once the heartbeat returns, the agent and node will automatically exit offline. Confirm recheck and make sure the node is healthy.
6

When escalate

If L1 falls, too, it’s not just the agent, it’s the node’s inaccessibility: go to the runbook of the node. If the agent doesn’t go up on a live node, collect a support bundle and escalate.
Remember: by themselves offline agents do not make a global dashboard critical - a maximum of degraded. The real inaccessibility of the node will independently show its reachability tests.

Runbook: the degradation of the region (external samples)

1

Symptom

External regional samples (L5) report the node being unavailable from the outside, even though it looks alive from the inside, the reason being external_regional_failure.
2

Diagnostics

Compare the picture: network from the inside (L1/L4 on the panel side) — okay? If yes, and the external sample drops — the problem is on the way between the region and the node (locking, routing the provider), not in the node itself. Check which blocks L5 is enabled for and what the different providers say. If no provider is connected, the status of external_probe_unavailable is not a failure.
3

Proposed action

Regional locking is usually not treated by action in the panel: the node is good, the path is not available. Recovery is at the level of cloaking/routing/infrastructure. Health here is a signal and a return confirmation, not a “fix” button.
4

Confirmation

Don’t restart the service “just in case” when everything is green inside: it won’t eliminate regional inaccessibility. Act for a reason, not a symptom.
5

Verification of recovery

Repeat the external sample for the affected block; make sure that the region sees the node again.
6

When escalate

Sustained regional degradation at a serviceable node is a pretext for camouflage/infrastructure handling and, if necessary, escalation.

Verification

After recovery, related action sentences are removed automatically, and the tape entry goes to auto_resolved.

Rollback

This page describes a process where there is nothing to roll back. The recoils of specific recovery actions are described where these actions are performed:
  • node withdrawal from the pool (drain_node) is canceled by action undrain_node - see Restoration;
  • re-application of the configuration and rollback of the schema version on the pages of Flow;
  • Node suppression is removed on the page Node suppression.

Typical errors

Keep order: network → service → dependencies. If L1 crashes, restarting the service won’t help - the problem is down the stack.
Check the preliminary plan and launch result. Re-use and rollback are available via Flow and Flow Guard; saving the image to a draft does not change the running route by itself.
As it should be: manual check makes one pass and does not expose critical from the node. Persistent failure fixes only scheduled check (multiple attempts).
Related pages: Reconstruction, dashboard health, Conditions and severity, Response to incidents, Node’s life cycle, Flow review.