> ## Documentation Index
> Fetch the complete documentation index at: https://docs.axottle.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbook

> Step-by-step scenarios for critical health alerts: critical node, critical circuit, inaccessible gateway, offline agent, and region degradation – from symptom to recovery check.

This page is a set of pre-made scripts (runbooks) for typical critical allerts. Each is arranged in the same way: **symptom → diagnosis → proposed action → confirmation → recovery check → when to escalate**. Keep the same order of diagnosis and do not fix the symptom at random.

> [!NOTE]
>
> For whom: panel administrator, SRE, duty engineer. Required rights: Administrator session.

> [!WARNING]
>
> This page describes manual operator scripts. [Flow Guard](https://docs.axottle.com/en/connection-units/recovery.md) allows you to confirm a specific launch or allow automatic plans for Flow in advance. Automation is limited to saved policy, fresh checks and limits.

## Diagnostic procedure

Always go from top to bottom - so you won't restart the service when the network actually crashed:

**Network → Service → Dependency → Data → Recovery**

1. **Network** Whether the node/gateway is available over the network (L1: reachability, DNS, ports).
2. **Service** - whether the server part is running (L2, requires an agent).
3. **Dependencies** - WARP, external proxies, CDNs, conditional connections and transit nodes.
4. **Data** - what the diagnostics, incident tapes and log show: what changed before the failure.
5. **Recovery** – just now choose and confirm the action.

## Runbook: node in critical

**1. Symptom**

The node is shown as `critical` on the dashboard, the reason is in the card: `entry_node_down`/`transit_node_down`, `remnanode_down`/`xray_down`, `disconnected` node in Remnawave or `offline` agent.

**2. Diagnostics**

Open the page of the node and its sample history (`GET .../nodes/{id}/probes`). In order: **net** — has passed the reachability (L1); **service** — what says L2 (needs an online agent); **Data** — collect logs through the action `view_remnanode_logs`/`view_xray_logs` (read only) and see the journal: did not coincide with the recent change.

**3. Proposed action**

The system will offer a suitable reason: `drain_node` (remove the node from the pool), `recheck` (if the agent is online), and restart the service with runtime (`restart_remnanode` or `restart_xray`). If the agent is offline, the restart buttons will not be available with a prompt-cause.

**4. Confirmation**

Confirm the action (`POST .../actions/{id}/confirm`) If users need to get off the problem node immediately, first `drain_node`, then deal with the service.

**5. Verification of recovery**

Run `POST .../nodes/{id}/check`; make sure reachability and service are back in `ok`, and consolidated node status is `healthy`.

**6. When escalate**

If, after restarting the service, the node falls again, or the agent does not communicate, collect a support bundle and escalate by [Response to incidents](https://docs.axottle.com/en/operations/incident-response.md).

## Runbook: Flow in Critical

**1. Symptom**

The chart is shown. `critical`Possible quorum causes: quorum of input nodes is lost (`<50%` healthy or 0 healthy), inaccessible gateway, persistent failure of the through test on the part of the node, drift configuration (`config_drift`).

**2. Diagnostics**

Open the diagram card and its diagnostics (`GET .../connection-units/{id}/diagnostics`) - fresh samples, associated log entries and metadata nodes. **network** (Are gateways and input nodes achievable?) **service** (Nodes respond to L2?) **dependency** (transit/routes/WARP/proxy) **data** (What has changed: configuration application, update, policy)

**3. Proposed action**

To the caus `config_drift` system will offer `reapply_config` at the schema level, and to critical - additionally `rollback_unit_version`. For specific nodes will be offers from runbook nodes (drain/restart/recheck).

**4. Confirmation**

> [!WARNING]
>
> Re-application and rollback are available through Flow's regular operations and Flow Guard plans. In the Version tab, the rollback uploads the picture to a draft and requires re-application; the recovery plan can perform the application itself. Before starting, check the steps shown and the impact on the draft. See [Application and versions of](https://docs.axottle.com/en/connection-units/lifecycle.md).

**5. Verification of recovery**

Run `POST .../connection-units/{id}/check` (one through passage). The circuit should return to `healthy`, and a series of unsuccessful checks should reset.

**6. When escalate**

If the quorum is not restored after repairing the nodes, or the drift of the configuration is repeated after repeated use, escalate with the application of diagnostics and journal.

## Runbook: Gateway unavailable

**1. Symptom**

Circuit in `critical` with the cause of the gateway inaccessibility (`gateway_unreachable`). As long as the gateway is not available, the input nodes are not important - the path for the user is severed.

**2. Diagnostics**

Start with **network**: whether the basic reachability of the gateway (L1 - DNS, TCP to the desired ports). Then **service** on the gateway node (L2), then **dependency** (whether the path is blocked by a firewall / provider).

**3. Proposed action**

If the gateway node itself is unavailable, use runbook nodes (Run the service with runtime, `recheck`). If the problem is networked and outside the node, the recovery is done on the infrastructure or provider side.

**4. Confirmation**

Confirm the service restart only if L2 shows that the service has fallen. If the network is external, eliminate it first, action in the panel will not help here.

**5. Verification of recovery**

`POST .../connection-units/{id}/check`: Gateway reachability - `ok`, circuit - `healthy`.

**6. When escalate**

If the gateway is available over the network, but the circuitry remains critical, it's not a gateway anymore, you can look at the runbook circuitry. If the inaccessibility is networked and out of your control, escalate to the provider/infrastructure.

## Runbook: Agent offline

**1. Symptom**

The agent of the node is shown `offline`; in the tape - incident by agent, in Telegram - category `agent_offline`. The agent is marked offline automatically after 90 seconds without heartbeat.

**2. Diagnostics**

Key question: is the node alive, but the agent is silent - or did the node really fall? **network**: Achievability tests (L1) come from the panel side and **dependency** If L1 is OK and L2 isn't available, the problem is probably the agent itself, not the node.

**3. Proposed action**

While the agent is offline, actions requiring the agent (restarts, log collection, WARP installation), **is not available** — gray buttons with a cause. `recheck` is available and, if necessary, to remove traffic, `drain_node`.

**4. Confirmation**

If the L1 sample is successful, it is usually enough to wait for the agent to return and perform `recheck`. Recovery of the agent as a service is performed on the node host (see [Node Life Cycle](https://docs.axottle.com/en/nodes/lifecycle.md)).

**5. Verification of recovery**

Once the heartbeat returns, the agent and node will automatically exit `offline`. Confirm `recheck` and make sure the node is `healthy`.

**6. When escalate**

If L1 falls, too, it's not just the agent, it's the node's inaccessibility: go to the runbook of the node. If the agent doesn't go up on a live node, collect a support bundle and escalate.

> [!NOTE]
>
> Remember: by themselves offline agents do not make a global dashboard `critical` - a maximum of `degraded`. The real inaccessibility of the node will independently show its reachability tests.

## Runbook: the degradation of the region (external samples)

**1. Symptom**

External regional samples (L5) report the node being unavailable from the outside, even though it looks alive from the inside, the reason being `external_regional_failure`.

**2. Diagnostics**

Compare the picture: **network** from the inside (L1/L4 on the panel side) — okay? If yes, and the external sample drops — the problem is on the way between the region and the node (locking, routing the provider), not in the node itself. Check which blocks L5 is enabled for and what the different providers say. If no provider is connected, the status of `external_probe_unavailable` is not a failure.

**3. Proposed action**

Regional locking is usually not treated by action in the panel: the node is good, the path is not available. Recovery is at the level of cloaking/routing/infrastructure. Health here is a signal and a return confirmation, not a "fix" button.

**4. Confirmation**

Don't restart the service "just in case" when everything is green inside: it won't eliminate regional inaccessibility. Act for a reason, not a symptom.

**5. Verification of recovery**

Repeat the external sample for the affected block; make sure that the region sees the node again.

**6. When escalate**

Sustained regional degradation at a serviceable node is a pretext for camouflage/infrastructure handling and, if necessary, escalation.

## Verification

After recovery, related action sentences are removed automatically, and the tape entry goes to `auto_resolved`.

## Rollback

This page describes a process where there is nothing to roll back. The recoils of specific recovery actions are described where these actions are performed:

* node withdrawal from the pool (`drain_node`) is canceled by action `undrain_node` - see [Restoration](https://docs.axottle.com/en/health/self-heal.md);
* re-application of the configuration and rollback of the schema version on the pages of [Flow](https://docs.axottle.com/en/connection-units/overview.md);
* Node suppression is removed on the [page Node](https://docs.axottle.com/en/health/suppression.md) suppression.

## Typical errors

> [accordion] Relaunched the service and the reason was online.
>
> Keep order: network → service → dependencies. If L1 crashes, restarting the service won't help - the problem is down the stack.

> [accordion] I was waiting for reapply\_config/rollback to fix the scheme.
>
> Check the preliminary plan and launch result. Re-use and rollback are available via Flow and Flow Guard; saving the image to a draft does not change the running route by itself.

> [accordion] Running recheck, and it doesn't escalate to critical
>
> As it should be: manual check makes one pass and does not expose critical from the node. Persistent failure fixes only scheduled check (multiple attempts).

Related pages: [Reconstruction](https://docs.axottle.com/en/health/self-heal.md), [dashboard health](https://docs.axottle.com/en/health/dashboard.md), [Conditions and severity](https://docs.axottle.com/en/health/states.md), [Response to incidents](https://docs.axottle.com/en/operations/incident-response.md), [Node's life cycle](https://docs.axottle.com/en/nodes/lifecycle.md), [Flow review](https://docs.axottle.com/en/connection-units/overview.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.