Operations and recovery

Operations is the practice of turning uncertain symptoms into evidence, containing impact, and leaving the system easier to understand next time.

The first five minutes

date -Is; uptime
systemctl --failed
free -h
df -hT
sudo ss -tulpn
sudo journalctl -p warning --since '-15 min' --no-pager

Then narrow to the affected service:

systemctl status cubis-api --no-pager
journalctl -u cubis-api --since '-15 min' --no-pager
curl -fsS -w '\n%{http_code} %{time_total}s\n' http://127.0.0.1:3000/health
sudo tail -n 100 /var/log/nginx/error.log
docker compose ps
docker compose logs --since=15m app

Follow each signal

Useful drill-down commands:

ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head
vmstat 1 5
sudo dmesg -T | tail -n 100
sudo journalctl -k | grep -i 'oom\|killed process'
sudo lsof +L1                     # deleted files still consuming disk
docker system df

Keep journals bounded

[Journal]
SystemMaxUse=1G
MaxRetentionSec=14day
Compress=yes

sudo systemctl restart systemd-journald
journalctl --disk-usage
sudo logrotate --debug /etc/logrotate.conf

Tune retention to incident and compliance needs. Logs required for investigations should be shipped off-host; a failed or compromised server cannot be its only evidence store.

Back up state, not machines

Define the recoverable components:

• database dumps or storage-native backups;
• user uploads and other persistent volumes;
• configuration represented as code;
• encrypted secrets in an approved secrets system; and
• the exact application image or release artifact.

Example PostgreSQL logical backup:

install -d -m 700 /var/backups/cubis
sudo -u postgres pg_dump -Fc cubis > /var/backups/cubis/cubis-$(date +%F).dump
sha256sum /var/backups/cubis/*.dump > /var/backups/cubis/SHA256SUMS

Copy backups to a separate account or region, encrypt them, apply retention, and monitor the job. A local file on the same server is not disaster recovery.

Test the restore

At a regular cadence, restore into an isolated environment:

Write down what happened

End the incident with a concise record: impact, start/end time, detection, timeline, contributing conditions, mitigations, recovery evidence, owner, and follow-up work. Separate learning from blame.