Signs: outbound traffic you cannot explain, an abuse notice from us, CPU pinned
by a process you do not recognise, unfamiliar SSH keys in
authorized_keys, cron entries you did not add, or a login from an address you
do not know.
1. Isolate it, do not power it off yet
Powering off destroys everything in memory, which is often the only evidence of what happened. Cut the traffic instead, using the cloud firewall:
- inbound policy
DROP - outbound policy
DROP
Then use the browser console, which does not need the network.
2. Look, before you change anything
# Who is logged in, and who has been
w
last -20
# What is running and what it is talking to
ps auxf
sudo ss -tulpn
# Keys that should not be there
cat ~/.ssh/authorized_keys /root/.ssh/authorized_keys 2>/dev/null
# Scheduled persistence
sudo crontab -l; ls -la /etc/cron.*; sudo systemctl list-timers --all
# Recently modified binaries
sudo find /usr/bin /usr/sbin /usr/local/bin -mtime -14 -type f
Copy anything you want to keep off the server now, to somewhere you are not going to execute it.
3. Rebuild. Do not clean.
A compromised server cannot be trusted again
Once something has had root, you cannot prove you found all of it. Rootkits hide from the tools you would use to look. The only safe end state is a fresh server, built from a known good configuration, with your data restored from a copy you trust.
Reinstall keeps the address and wipes the disk, which is usually what you want. A brand new server is cleaner still if you can move the DNS.
Rebuild it hardened this time: keys only, no password auth, default-deny inbound. See Harden a new server.
4. Rotate everything the server could see
Every credential that was on it is burned: database passwords, API keys for
other services, TLS private keys, SSH keys whose private half was stored there,
.env files. Rotate them all, not just the ones you think were read.
5. Tell us if there is abuse traffic
If we sent you an abuse notice, reply on that ticket with what you found and what you have done. If you found it first, open a ticket. Outbound attack traffic gets a server suspended under the Acceptable Use Policy, and it is much better if you are already in the conversation.
Preventing the next one
Almost every compromise we see is one of four things: password SSH left on, a
database bound to 0.0.0.0, an unpatched application, or a credential committed
to a public repository. The first two are ten minutes of work.