Writing an Incident Runbook for a Small VPS Team

A runbook should help an authorized operator make the next good decision under pressure. It does not need to describe every technical detail of the application. For a small team operating a few VPS instances, a short document tied to a specific symptom is usually easier to keep accurate than an enormous collection of untested commands.

Define the entry condition and service owner

Name the symptom the runbook addresses, such as repeated public HTTP failures or a backup older than the permitted recovery window. Identify the service, environment, owner, and escalation contact. Include the expected customer impact so the operator understands urgency. Google's on-call workbook chapter discusses the preparation and support needed by responders. Adapt that idea to your team's actual availability rather than assigning an imaginary around-the-clock rotation nobody can staff.

Link to access without embedding secrets

Explain where authorized operators obtain SSH access, console access, application credentials, and backup recovery keys. Link to the protected system that manages them instead of copying the values into the runbook. Make sure the document remains accessible if the application VPS is down. Record which operations require additional authority, particularly restoring production data or changing public network access. An operator should not have to guess whether a destructive action is part of their role during an outage.

Write diagnostic branches from observations

Use a few explicit conditions: if the public check fails but the local application responds, inspect the public request path; if both fail and the database is unavailable, inspect the dependency; if the guest is unreachable, use console and platform observations. Each branch should name a bounded check and explain what its result means. Avoid long command chains whose output must be interpreted by the original author. Include expected successful behavior and the location of detailed reference documentation.

Constrain mitigations and define rollback

For each approved mitigation, state the exact target, expected effect, possible interruption, and stop condition. A runbook saying restart everything provides little protection against making an incident worse. A more useful entry names the affected worker service, asks the operator to record its failure reason, and describes how to confirm that processing resumes without duplicates. Link to the tested recovery procedure for data restoration. Specify when conflicting evidence or repeated failure requires escalation instead of another identical retry.

Test it with someone who did not write it

Run a tabletop exercise or a safe staging incident with another operator following the document. Note unclear service names, missing permissions, broken links, and assumptions about local shell setup. Time how long it takes to reach the first useful observation. Update the runbook immediately from those findings and record its last exercise date. After a real incident, add the new diagnostic distinction that would have shortened the response, while removing outdated steps that no longer apply.

Use LayerOne's help center for platform procedures and link to the backup responsibility guide where recovery is discussed. A good runbook earns trust through successful use by another operator, with clear boundaries and visible evidence for every important decision.