```
Create a recovery branch at the commit before changing anything else. Once the files are verified, cherry-pick or merge the recovered work into the correct branch.
Recovery rule
Stop, inspect, and create a branch or tag before trying another history-changing command. Repeated resets often make the original problem harder to understand.
## References [#references]
* [git-restore](https://git-scm.com/docs/git-restore)
* [git-revert](https://git-scm.com/docs/git-revert)
* [git-reflog](https://git-scm.com/docs/git-reflog)
* [git-stash](https://git-scm.com/docs/git-stash)
---
# Server access and security (/docs/linux/access-and-hardening)
Category: Security
Level: Intermediate
Tags: ssh, hardening, ufw, fail2ban, sudo
Last reviewed: 2026-08-13
Secure access is the first production task. Complete it before deploying an application or placing data on the server.
## 1. Create and install an SSH key [#1-create-and-install-an-ssh-key]
Run key generation on your workstation:
```bash
ssh-keygen -t ed25519 -a 64 -C 'you@cubis'
ssh-copy-id root@203.0.113.10
ssh root@203.0.113.10
```
Protect the private key with a passphrase and never copy it to the server. The `.pub` file is safe to distribute.
## 2. Create a named operator [#2-create-a-named-operator]
```bash
adduser deploy
usermod -aG sudo deploy
install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys
```
Open a **second terminal** and verify `ssh deploy@203.0.113.10` plus `sudo -v`. Keep the root session open until all access changes work.
## 3. Harden the SSH daemon [#3-harden-the-ssh-daemon]
Create an included config instead of rewriting the vendor file:
```ini title="/etc/ssh/sshd_config.d/10-cubis-hardening.conf"
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
AllowUsers deploy
```
```bash
sudo sshd -t # syntax check: no output means valid
sudo systemctl reload ssh # Ubuntu/Debian service name
ssh -o PreferredAuthentications=publickey deploy@203.0.113.10
```
A non-default port reduces log noise, not the need for key authentication. If you change it, allow the new port in the cloud firewall and UFW before reloading SSH.
## 4. Apply a default-deny firewall [#4-apply-a-default-deny-firewall]
Cloud firewalls and host firewalls solve different problems; use both when available.
```bash
sudo apt install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status numbered
```
Do not expose database ports such as `5432`, `3306`, `6379`, or `27017` to the public internet. Bind them to loopback, a private interface, or a private network.
## 5. Reduce brute-force noise [#5-reduce-brute-force-noise]
```bash
sudo apt install fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd
```
Fail2ban adds protection, but it does not replace SSH keys or a firewall. Check that its SSH rule reads the correct systemd journal or log for your distribution.
## 6. Keep security updates moving [#6-keep-security-updates-moving]
```bash
sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades
systemctl status unattended-upgrades
```
Define how reboots are scheduled; unattended package installation does not guarantee that a new kernel is running.
## Before using the server in production [#before-using-the-server-in-production]
* [ ] A named human identity can connect with a passphrase-protected key.
* [ ] Root and password SSH login are disabled and verified in a new session.
* [ ] Inbound traffic defaults to deny.
* [ ] Only justified public ports exist in cloud and host firewalls.
* [ ] Application and database processes run without root.
* [ ] Security updates and reboot ownership are defined.
* [ ] Provider console or recovery access was tested.
Record what you changed
Save the results of `sshd -T`, `ufw status verbose`, and `ss -tulpn` with the server handoff. Include the update and reboot policy. Never include private keys or secrets.
---
# Deploying applications (/docs/linux/deployment)
Category: Application delivery
Level: Advanced
Tags: docker, compose, nginx, tls, deployment
Last reviewed: 2026-08-13
This guide uses Docker Compose for the application stack and host-managed Nginx for the public edge. The same boundaries apply if you run the app directly with systemd.
## Install Docker from its official repository [#install-docker-from-its-official-repository]
Use the current instructions for your distribution and verify the repository fingerprint before installation. Afterward:
```bash
docker version
docker compose version
sudo systemctl enable --now docker
sudo usermod -aG docker deploy
```
Log out and back in for group membership to refresh. Membership in the `docker` group is effectively root-equivalent; grant it only to trusted operators.
## Build a small, non-root image [#build-a-small-non-root-image]
```dockerfile title="Dockerfile"
FROM node:22-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci
FROM node:22-alpine AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build && npm prune --omit=dev
FROM node:22-alpine AS runtime
ENV NODE_ENV=production
WORKDIR /app
USER node
COPY --chown=node:node --from=build /app ./
EXPOSE 3000
CMD ["node", "server.js"]
```
Pin a known runtime major or immutable digest, exclude secrets with `.dockerignore`, and scan the final image in CI.
## Define the runtime [#define-the-runtime]
```yaml title="compose.yaml"
services:
app:
image: registry.example.com/cubis-api:${APP_VERSION}
restart: unless-stopped
env_file: /etc/cubis-api.env
ports:
- "127.0.0.1:3000:3000"
healthcheck:
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 20s
security_opt:
- no-new-privileges:true
read_only: true
tmpfs:
- /tmp
```
Avoid `latest`. Deploy an immutable commit SHA or release version so the running artifact and rollback target are unambiguous.
```bash
export APP_VERSION=2026.08.13-3f28c1a
docker compose pull
docker compose config --quiet
docker compose up -d --remove-orphans
docker compose ps
docker compose logs --tail=100 app
```
## Put Nginx in front [#put-nginx-in-front]
```nginx title="/etc/nginx/sites-available/cubis-api"
server {
listen 80;
listen [::]:80;
server_name api.example.com;
location / {
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
}
}
```
```bash
sudo ln -s /etc/nginx/sites-available/cubis-api /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
curl -I -H 'Host: api.example.com' http://127.0.0.1
```
## Enable TLS [#enable-tls]
Point DNS at the server first, allow ports 80 and 443, then use your organization’s certificate automation. With Certbot on Ubuntu:
```bash
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d api.example.com
sudo certbot renew --dry-run
systemctl list-timers | grep certbot
```
## Release and rollback [#release-and-rollback]
### Record the intended version [#record-the-intended-version]
Link the change, image digest, migration plan, owner, verification, and rollback target.
### Pull before changing runtime [#pull-before-changing-runtime]
`docker compose pull` makes registry failures happen before the current containers are replaced.
### Treat database migrations separately [#treat-database-migrations-separately]
Make database changes in stages: add the new structure, move the data, and remove the old structure only after every running application version has stopped using it. Know whether the migration can be reversed before deploying it.
### Start and verify [#start-and-verify]
Check container health, local health endpoint, public HTTPS, key user flow, and logs.
### Roll back deliberately [#roll-back-deliberately]
Set `APP_VERSION` to the last known-good immutable version and run `docker compose up -d`. Verify with the same checklist.
A single container must release its port before the replacement can use it. To avoid that interruption, run at least two healthy instances behind a load balancer, add readiness checks, and keep database changes compatible with both application versions.
---
# Linux foundations (/docs/linux/foundations)
Category: Cloud & infrastructure
Level: Foundation
Tags: linux, shell, filesystem, packages, permissions
Last reviewed: 2026-08-13
Linux exposes almost everything as files, processes, sockets, and users. Learn those four ideas and server work becomes much less mysterious.
## Inspect an unfamiliar server [#inspect-an-unfamiliar-server]
Run these before making changes:
```bash
whoami # current identity
hostnamectl # OS, kernel, hostname, virtualization
uname -a # kernel and architecture
cat /etc/os-release # distribution details
uptime # runtime and load averages
free -h # memory and swap
df -hT # mounted filesystems and capacity
```
The three load-average values represent runnable or uninterruptible tasks over 1, 5, and 15 minutes. Compare them with the number of CPU cores from `nproc`; a sustained load far above the core count deserves investigation.
## Know the filesystem [#know-the-filesystem]
| Path | Operational purpose |
| -------------------- | ------------------------------------------ |
| `/etc` | System and service configuration |
| `/var/log` | Persistent logs |
| `/var/lib` | Service-owned persistent state |
| `/var/www` or `/srv` | Common application locations |
| `/home` | Human user directories |
| `/opt` | Self-contained third-party software |
| `/run` | Runtime state cleared at boot |
| `/tmp` | Temporary files; do not assume persistence |
```bash
pwd # current directory
ls -lah # include hidden files and readable sizes
cd /var/www
mkdir -p cubis/releases
find /var/log -type f -name '*.log'
du -sh /var/* 2>/dev/null | sort -h
```
Do not delete unfamiliar files at random. Find the full filesystem with `df -hT`, locate large directories with `du` or `ncdu`, then check whether logs, container layers, package caches, or deleted-open files are responsible.
## Read and change text safely [#read-and-change-text-safely]
```bash
less /var/log/syslog # scroll; / searches; q exits
tail -n 100 app.log # last 100 lines
tail -F app.log # follow across log rotation
grep -Rin 'connection refused' /var/log
cp nginx.conf nginx.conf.bak # backup before editing
sudoedit /etc/nginx/nginx.conf # edit with your own editor identity
diff -u nginx.conf.bak nginx.conf
```
Use `rm -rf` only after resolving the exact path. It bypasses the trash and recursively removes entries without normal confirmation.
## Identify the Linux family [#identify-the-linux-family]
Do not guess the distribution from a cloud provider or image name. Read the operating-system metadata first:
```bash
cat /etc/os-release
command -v apt dnf yum apk zypper
systemctl --version
```
`ID` identifies the distribution and `ID_LIKE` lists related families. Package tools, package names, service names, firewall defaults, and security systems can differ even when the shell commands look familiar.
## Install and update packages [#install-and-update-packages]
```bash
sudo apt update # refresh package information
apt list --upgradable # show available upgrades
sudo apt upgrade # install normal upgrades
sudo apt install nginx git curl jq htop # install named packages
apt show nginx # package details
dpkg -L nginx # files installed by the package
sudo apt autoremove # remove unused dependencies
```
`apt update` only refreshes package information. It does not install upgrades.
```bash
sudo dnf makecache # refresh package information
dnf check-update # show available upgrades
sudo dnf upgrade # install upgrades
sudo dnf install nginx git curl jq htop # install named packages
dnf info nginx # package details
rpm -ql nginx # files installed by the package
sudo dnf autoremove # remove unused dependencies
```
Rocky Linux, AlmaLinux, and current RHEL releases normally use `dnf`. Older releases may expose `yum` as a compatibility command.
```bash
sudo dnf makecache # refresh package information
dnf check-update # show available upgrades
sudo dnf upgrade # install upgrades
sudo dnf install nginx git curl jq htop # install named packages
dnf info nginx # package details
rpm -ql nginx # files installed by the package
sudo dnf autoremove # remove unused dependencies
```
Amazon Linux 2023 uses `dnf`. Check `/etc/os-release` rather than assuming commands from Amazon Linux 2 apply to a newer image.
Test major upgrades and kernel changes before production rollout. After installing a new kernel, plan and verify the reboot instead of assuming the running kernel changed.
## Common family differences [#common-family-differences]
| Task | Ubuntu / Debian | RHEL / Rocky / Amazon Linux |
| --------------- | ------------------------------------ | -------------------------------------- |
| Packages | `apt`, `dpkg` | `dnf`, `rpm` |
| SSH service | usually `ssh` | usually `sshd` |
| Host firewall | commonly UFW | commonly firewalld |
| Security policy | AppArmor on Ubuntu | SELinux on RHEL-family systems |
| Main system log | journal; sometimes `/var/log/syslog` | journal; sometimes `/var/log/messages` |
Use `systemctl list-unit-files | grep -E 'ssh|sshd'` and `systemctl status NAME` to confirm the actual service name. Do not disable SELinux or AppArmor just to make an application work; read the denial and fix the policy or file context.
## Users, groups, and permissions [#users-groups-and-permissions]
An `ls -l` entry separates permissions for the owner, group, and everyone else:
```text
-rwxr-xr-- 1 deploy www-data 2048 app.sh
│└┬┘└┬┘└┬┘
│ │ │ └─ others: read
│ │ └──── group: read + execute
│ └─────── owner: read + write + execute
└───────── regular file
```
```bash
id # UID, primary GID, supplementary groups
getent group www-data
chmod 750 deploy.sh # owner rwx, group r-x, others none
chmod 600 .env # owner read/write only
chown deploy:www-data /srv/app
sudo -l # commands this identity may elevate
```
Avoid `chmod 777`. If a process cannot write, first identify its user with `systemctl show -p User SERVICE` or `ps`, then set deliberate ownership and the narrowest useful permissions.
## Shell composition [#shell-composition]
```bash
command-a | command-b # pipe stdout into another command
command > file # replace file with stdout
command >> file # append stdout
command 2> errors.log # redirect stderr
command && next # run next only after success
command || recovery # run recovery only after failure
```
Check your understanding
Given any file, you can identify its owner, group, permissions, filesystem, package source, and the process currently using it.
---
# Linux for Cloud Engineers (/docs/linux)
Category: Cloud & infrastructure
Level: Foundation
Tags: linux, cloud, devops, deployment, operations
Last reviewed: 2026-08-13
This guide is for engineers who build applications and also need to manage the servers that run them. The examples use Ubuntu and Debian. Other distributions use many of the same tools, but package names and service configuration can differ.
You will set up access, remove unsafe defaults, run an application as a service or container, add HTTPS, investigate failures, and restore data from a backup.
## How a request reaches your application [#how-a-request-reaches-your-application]
```text
Internet
│
DNS → firewall → Nginx / TLS
│
app service :3000
│
database / storage
│
logs + monitoring
```
Public traffic enters through the firewall and Nginx. Nginx handles HTTPS and sends the request to the application. The application runs without root access, and the database is not exposed to the public internet. Logs help you investigate problems; backups let you recover data.
## Learning path [#learning-path]
1 · Foundation
Linux foundations
Filesystem, shell, packages, files, users, and permissions.
2 · Intermediate
Server access and security
SSH keys, deploy users, UFW, fail2ban, and safe changes.
3 · Intermediate
Services and networking
systemd, processes, ports, DNS, HTTP, and diagnostics.
4 · Advanced
Application deployment
Docker Compose, Nginx, TLS, releases, and rollback.
5 · Advanced
Operations and recovery
Logs, health, disk pressure, incidents, and backups.
6 · Reference
Command reference
Commands for daily server work and troubleshooting.
## Before you begin [#before-you-begin]
Use a disposable test server—not production—for the first pass. You need:
* a recent Ubuntu LTS or Debian server;
* its public IP and provider console access;
* a local terminal with OpenSSH;
* a domain name for the deployment section; and
* a Git repository containing a small web app.
## How to work safely [#how-to-work-safely]
1. **Observe before changing.** Capture current state and the expected outcome.
2. **Prefer reversible changes.** Back up config, validate it, then reload—not restart—when possible.
3. **Keep one recovery path open.** Provider console access and a second SSH session prevent avoidable lockouts.
4. **Run applications without root.** Grant only the access a service actually needs.
5. **Automate repeated work.** First understand the task; then document and automate it.
Before using this in production
Make sure you can explain every public port, identify the process behind it, find its logs, deploy a specific version, and restore its data on another server.
---
# Operations and recovery (/docs/linux/operations)
Category: Cloud & infrastructure
Level: Advanced
Tags: monitoring, logs, incidents, backups, recovery
Last reviewed: 2026-08-13
Operations is the practice of turning uncertain symptoms into evidence, containing impact, and leaving the system easier to understand next time.
## The first five minutes [#the-first-five-minutes]
```bash
date -Is; uptime
systemctl --failed
free -h
df -hT
sudo ss -tulpn
sudo journalctl -p warning --since '-15 min' --no-pager
```
Then narrow to the affected service:
```bash
systemctl status cubis-api --no-pager
journalctl -u cubis-api --since '-15 min' --no-pager
curl -fsS -w '\n%{http_code} %{time_total}s\n' http://127.0.0.1:3000/health
sudo tail -n 100 /var/log/nginx/error.log
docker compose ps
docker compose logs --since=15m app
```
Record the time, commands, results, and changes. Read the error first. Clearing logs or repeatedly restarting a service can remove the information you need to find the cause.
## Follow each signal [#follow-each-signal]
| Signal | Ask next |
| ---------------- | ------------------------------------------------------------------- |
| CPU saturated | Which process? Is work expected? Is load runnable or I/O blocked? |
| Memory low | Is swap active? Is the kernel killing processes? Is usage growing? |
| Disk full | Which filesystem and directory? Are deleted files still open? |
| 502 from Nginx | Is the upstream listening? Is its health endpoint healthy? |
| Timeout | DNS, route, firewall, listener, application, or dependency latency? |
| Frequent restart | What exit code and journal message preceded it? |
Useful drill-down commands:
```bash
ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head
vmstat 1 5
sudo dmesg -T | tail -n 100
sudo journalctl -k | grep -i 'oom\|killed process'
sudo lsof +L1 # deleted files still consuming disk
docker system df
```
## Keep journals bounded [#keep-journals-bounded]
```ini title="/etc/systemd/journald.conf.d/10-limits.conf"
[Journal]
SystemMaxUse=1G
MaxRetentionSec=14day
Compress=yes
```
```bash
sudo systemctl restart systemd-journald
journalctl --disk-usage
sudo logrotate --debug /etc/logrotate.conf
```
Tune retention to incident and compliance needs. Logs required for investigations should be shipped off-host; a failed or compromised server cannot be its only evidence store.
## Back up state, not machines [#back-up-state-not-machines]
Define the recoverable components:
* database dumps or storage-native backups;
* user uploads and other persistent volumes;
* configuration represented as code;
* encrypted secrets in an approved secrets system; and
* the exact application image or release artifact.
Example PostgreSQL logical backup:
```bash
install -d -m 700 /var/backups/cubis
sudo -u postgres pg_dump -Fc cubis > /var/backups/cubis/cubis-$(date +%F).dump
sha256sum /var/backups/cubis/*.dump > /var/backups/cubis/SHA256SUMS
```
Copy backups to a separate account or region, encrypt them, apply retention, and monitor the job. A local file on the same server is not disaster recovery.
## Test the restore [#test-the-restore]
At a regular cadence, restore into an isolated environment:
### Start from empty infrastructure [#start-from-empty-infrastructure]
Provision a fresh server or isolated database. Do not rely on undocumented remnants.
### Fetch and verify backup integrity [#fetch-and-verify-backup-integrity]
Validate checksums and decryption before attempting restore.
### Restore data and deploy the matching application [#restore-data-and-deploy-the-matching-application]
Record tool versions and duration; watch for schema incompatibility.
### Run functional checks [#run-functional-checks]
Validate representative records, authentication, writes, uploads, and critical user journeys.
### Record achieved RPO and RTO [#record-achieved-rpo-and-rto]
Recovery point objective is tolerable data loss. Recovery time objective is tolerable outage. The test reveals whether you actually meet them.
## Write down what happened [#write-down-what-happened]
End the incident with a concise record: impact, start/end time, detection, timeline, contributing conditions, mitigations, recovery evidence, owner, and follow-up work. Separate learning from blame.
A backup is not enough
Test the restore on a schedule. The test should show that the team can recover working data and service within the agreed recovery time.
---
# Linux command reference (/docs/linux/quick-reference)
Category: Cloud & infrastructure
Level: Reference
Tags: linux, commands, cheatsheet, troubleshooting
Last reviewed: 2026-08-13
Use this page after you understand the commands. Copying an unfamiliar destructive or privileged command into production is not an operating procedure.
## System state [#system-state]
```bash
hostnamectl # system identity
uptime # uptime and load
free -h # memory and swap
df -hT # filesystem capacity and type
du -sh * 2>/dev/null | sort -h # size of entries here
ps aux --sort=-%cpu | head # top CPU processes
systemctl --failed # failed units
timedatectl # clock, timezone, and time sync
last reboot # recent reboot history
dmesg --level=err,warn # current boot's kernel warnings
```
## Package management by family [#package-management-by-family]
```bash
sudo apt update # refresh package information
apt list --upgradable # list available upgrades
sudo apt upgrade # install normal upgrades
sudo apt install PACKAGE # install a package
apt show PACKAGE # inspect package metadata
dpkg -S /path/to/file # find the package owning a file
```
```bash
sudo dnf makecache # refresh package information
dnf check-update # list available upgrades
sudo dnf upgrade # install upgrades
sudo dnf install PACKAGE # install a package
dnf info PACKAGE # inspect package metadata
rpm -qf /path/to/file # find the package owning a file
```
```bash
sudo dnf makecache # Amazon Linux 2023
dnf check-update
sudo dnf upgrade
sudo dnf install PACKAGE
dnf info PACKAGE
rpm -qf /path/to/file
```
## Files and text [#files-and-text]
```bash
ls -lah # detailed listing
find /path -type f -name '*.log' # find by name and type
less file # inspect safely
tail -F file # follow a rotating log
grep -Rin 'pattern' /path # recursive content search
cp -a source destination # preserve metadata when copying
rsync -a --dry-run src/ dest/ # preview synchronization
namei -l /full/path/to/file # permissions on every path component
file archive.tar.gz # detect file type from content
sha256sum file # calculate an integrity checksum
```
## Archives and transfers [#archives-and-transfers]
```bash
tar -czf backup.tar.gz directory/ # create a gzip-compressed archive
tar -tzf backup.tar.gz # list archive contents before extraction
tar -xzf backup.tar.gz # extract the archive here
rsync -aHAX --dry-run src/ dest/ # preview a metadata-preserving local copy
scp file user@host:/tmp/ # copy one file over SSH
sftp user@host # interactive transfer over SSH
```
Inspect an archive before extracting it as root. For full server copies, confirm filesystem, ACL, extended-attribute, and hard-link requirements before choosing `rsync` flags.
## Identities and permissions [#identities-and-permissions]
```bash
whoami
id
sudo -l
stat file
chmod 640 file
chown user:group file
getent passwd deploy
getent group www-data
getent hosts example.com
sudo -u deploy command # run one command as the service user
umask # default permissions for new files
```
## Processes and scheduled work [#processes-and-scheduled-work]
```bash
pgrep -af nginx # find matching PIDs and full commands
ps -o pid,ppid,user,%cpu,%mem,etime,cmd -p PID
sudo lsof -p PID # files and sockets opened by a process
nice -n 10 command # start with lower CPU scheduling priority
renice 10 -p PID # adjust a running process priority
systemctl list-timers --all # systemd timers and their next runs
crontab -l # current user's cron jobs
sudo crontab -u deploy -l # another user's cron jobs
```
## Services and logs [#services-and-logs]
```bash
systemctl status SERVICE --no-pager
systemctl restart SERVICE
systemctl reload SERVICE
systemctl enable --now SERVICE
journalctl -u SERVICE -n 100 --no-pager
journalctl -u SERVICE -f
journalctl --since '-30 min' -p warning
systemctl cat SERVICE # unit file plus drop-in configuration
systemctl show SERVICE -p User -p MainPID -p ExecStart
journalctl -b -u SERVICE # service logs from the current boot
```
## Network and HTTP [#network-and-http]
```bash
ip -brief address
ip route
sudo ss -tulpn
dig +short example.com A
nc -vz example.com 443
curl -fsS http://127.0.0.1:3000/health
curl -vI https://example.com
sudo ufw status numbered
resolvectl status # configured DNS resolvers
ip route get 1.1.1.1 # route chosen for a destination
openssl s_client -connect example.com:443 -servername example.com
```
For RHEL-family firewalls, inspect `sudo firewall-cmd --list-all` instead of UFW. A successful TCP connection does not prove the application is healthy; follow it with an HTTP or protocol-specific check.
## Storage and disk pressure [#storage-and-disk-pressure]
```bash
lsblk -f # block devices, filesystems, and mount points
findmnt # mounted filesystems and options
df -hT # capacity by filesystem
df -ih # inode usage
du -xhd1 /var | sort -h # directory sizes on one filesystem
sudo lsof +L1 # deleted files still held open
journalctl --disk-usage # journal storage use
```
When `df` reports a full disk but `du` cannot account for the space, look for deleted-open files with `lsof +L1` and check container storage.
## Containers [#containers]
```bash
docker compose ps
docker compose logs -f --tail=100 app
docker compose pull
docker compose up -d --remove-orphans
docker compose exec app sh
docker inspect CONTAINER
docker system df
docker stats --no-stream
docker inspect --format '{{json .State.Health}}' CONTAINER
```
## Security checks [#security-checks]
```bash
sudo ss -lntup # listening ports and owning processes
sudo last -a | head # recent login history
sudo journalctl -u ssh --since today # Ubuntu / Debian SSH events
sudo journalctl -u sshd --since today # RHEL-family SSH events
sudo find / -xdev -perm -4000 -type f # setuid files on the root filesystem
sudo getenforce # SELinux mode, when installed
sudo aa-status # AppArmor status, when installed
```
Treat this output as evidence to review, not a pass/fail security scan. Compare it with the server's intended ports, users, and policy.
## Where to start [#where-to-start]
| Symptom | First commands |
| ---------------------- | ---------------------------------------------------------- |
| Site unavailable | `dig +short`, `nc -vz HOST 443`, `curl -vI` |
| Nginx 502 | `ss -ltnp`, local `curl`, Nginx error log, service journal |
| SSH timeout | provider status, cloud firewall, route, UFW, SSH listener |
| Permission denied | `id`, `namei -l PATH`, `stat`, service user |
| Disk full | `df -hT`, `du`, `lsof +L1`, `journalctl --disk-usage` |
| Process disappeared | `systemctl status`, `journalctl -u`, kernel OOM log |
| Container restart loop | `docker compose ps`, `logs`, `inspect`, health check |
## Safe configuration change [#safe-configuration-change]
```bash
sudo cp -a config config.bak.$(date +%Y%m%d%H%M%S)
sudoedit config
VALIDATOR -t # nginx -t, sshd -t, compose config --quiet
sudo systemctl reload SERVICE
systemctl status SERVICE --no-pager
```
For `rm`, `chmod`, `chown`, firewall rules, database changes, and recursive operations: resolve the target, preview where possible, keep a recovery path, and know how to verify success.
## Continue learning [#continue-learning]
* Return to [Linux foundations](./foundations) for command semantics.
* Use [Services and networking](./services-and-networking) to trace traffic.
* Follow [Operations and recovery](./operations) during an incident.
---
# Services and networking (/docs/linux/services-and-networking)
Category: Cloud & infrastructure
Level: Intermediate
Tags: systemd, networking, dns, ports, processes
Last reviewed: 2026-08-13
Most production failures become tractable when you can answer two questions: **what process should be running?** and **how should traffic reach it?**
## Inspect processes and resources [#inspect-processes-and-resources]
```bash
ps aux --sort=-%mem | head
pgrep -af 'node|python|java'
top # press 1 for per-CPU view
systemctl --failed
kill -TERM 1234 # request graceful shutdown
```
Prefer `SIGTERM` and wait for graceful shutdown. Use `SIGKILL` only when the process cannot respond; it prevents cleanup and can leave state inconsistent.
## Create a systemd service [#create-a-systemd-service]
```ini title="/etc/systemd/system/cubis-api.service"
[Unit]
Description=Cubis API
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/srv/cubis-api/current
EnvironmentFile=/etc/cubis-api.env
ExecStart=/usr/bin/node server.js
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
NoNewPrivileges=true
PrivateTmp=true
[Install]
WantedBy=multi-user.target
```
Keep secrets in a root-owned file such as `/etc/cubis-api.env` with mode `600`; do not store them in the unit or repository.
```bash
sudo systemd-analyze verify /etc/systemd/system/cubis-api.service
sudo systemctl daemon-reload
sudo systemctl enable --now cubis-api
systemctl status cubis-api --no-pager
journalctl -u cubis-api -n 100 --no-pager
```
After changing application code, restart the service. After changing only the unit, run `daemon-reload` first.
## Understand listening sockets [#understand-listening-sockets]
```bash
sudo ss -tulpn # TCP/UDP listeners and owning processes
sudo ss -ltnp 'sport = :3000'
curl -fsS http://127.0.0.1:3000/health
```
Binding to `127.0.0.1:3000` means only local processes can connect. Binding to `0.0.0.0:3000` exposes the socket on every IPv4 interface if the firewall allows it. An application behind Nginx should normally bind to loopback.
## Trace the request path [#trace-the-request-path]
Follow the layers in order instead of guessing:
### Resolve DNS [#resolve-dns]
`dig +short app.example.com A` should return the expected public IP. Also inspect `AAAA` if IPv6 is published.
### Reach the host [#reach-the-host]
`nc -vz app.example.com 443` checks TCP reachability. A timeout suggests routing or firewall; refusal means the host answered but nothing accepted the port.
### Negotiate TLS and HTTP [#negotiate-tls-and-http]
`curl -vI https://app.example.com` exposes DNS, connection, certificate, protocol, redirect, and response headers.
### Reach the application locally [#reach-the-application-locally]
On the server, `curl -v http://127.0.0.1:3000/health`. If local works but public fails, focus on Nginx, TLS, or the firewall.
### Correlate logs [#correlate-logs]
Read Nginx access/error logs and the service journal for the same timestamp or request ID.
## DNS and route tools [#dns-and-route-tools]
| Command | Use |
| ---------------------------------------- | ------------------------------------------ |
| `dig +short name A` | Resolve IPv4 records |
| `resolvectl query name` | Query through the host resolver |
| `ip -brief address` | Show interface addresses |
| `ip route` | Show routing decisions and default gateway |
| `tracepath host` | Discover path and MTU issues |
| `curl -w '%{http_code} %{time_total}\n'` | Measure HTTP result and total time |
Diagnostic habit
State the failing layer: “DNS resolves, TCP 443 connects, TLS succeeds, Nginx returns 502, and the local health check refuses port 3000.” That sentence is far more actionable than “the server is down.”
---
# Addressing and Routing (/docs/networking/addressing-and-routing)
Category: Cloud & infrastructure
Level: Foundation
Tags: networking, ip, cidr, routing, subnets
Last reviewed: 2026-08-13
An address identifies an interface. A prefix defines the network around it. A route selects the next hop and interface for a destination.
## Read a prefix [#read-a-prefix]
For `10.20.4.17/24`, the first 24 bits describe the network and the remaining 8 bits identify addresses inside it. The network prefix is `10.20.4.0/24`.
| Prefix | Total IPv4 addresses | Common use |
| ------ | -------------------: | ------------------------------- |
| `/32` | 1 | One host or route target |
| `/28` | 16 | Small subnet |
| `/24` | 256 | Conventional application subnet |
| `/16` | 65,536 | Larger private network boundary |
Cloud providers reserve addresses inside a subnet for platform functions. Do not calculate usable capacity from the total alone; check the provider’s subnet rules.
## Inspect interfaces and routes [#inspect-interfaces-and-routes]
```bash
ip -brief address
ip route show
ip -6 route show
ip rule show
```
A typical IPv4 route table might contain:
```text
default via 10.20.4.1 dev eth0
10.20.4.0/24 dev eth0 proto kernel scope link src 10.20.4.17
```
The connected route reaches the local subnet directly. The default route sends destinations without a more-specific match to the gateway. Linux selects the most specific matching prefix; route metric helps choose between otherwise comparable routes.
Ask the kernel how it would route one destination without sending a packet:
```bash
ip route get 203.0.113.10
ip -6 route get 2001:db8::10
```
Check the selected interface, gateway, and source address. If the source is wrong on a multi-homed host, inspect policy rules with `ip rule` as well as the main route table.
## Map the cloud path [#map-the-cloud-path]
A common layout separates public entry points from private workloads:
```text
Internet
└─ public load balancer
└─ private application subnet
└─ database subnet
private outbound traffic → NAT gateway or controlled egress proxy
```
For every subnet, document:
* its IPv4 and IPv6 prefixes;
* the route table attached to it;
* the path for internet, private network, and service endpoints;
* inbound and outbound firewall policy; and
* whether addresses are stable or allocated dynamically.
Route tables provide reachability; they do not grant permission. A valid route can still be blocked by a security group, network ACL, host firewall, or service binding.
## IPv6 changes the assumptions [#ipv6-changes-the-assumptions]
An IPv6 address can be globally routable without IPv4-style NAT. That does not mean it is publicly allowed: enforce explicit inbound and outbound policy and confirm the service binds to IPv6.
```bash
ip -6 address show scope global
ip -6 route
ss -6 -lntp
curl -6 -I https://example.com
```
Test IPv4 and IPv6 independently. A published `AAAA` record with a broken IPv6 path can produce intermittent-looking failures because clients may choose different address families.
Replacing an interface address or default route can end the SSH session immediately. Use a disposable lab first. On a remote server, keep provider-console access open and arrange an automatic rollback before applying the change.
## References [#references]
* [Classless inter-domain routing](https://www.rfc-editor.org/rfc/rfc4632)
* [IPv6 addressing architecture](https://www.rfc-editor.org/rfc/rfc4291)
* [`ip-route` manual](https://man7.org/linux/man-pages/man8/ip-route.8.html)
---
# DNS, HTTP, and TLS (/docs/networking/dns-http-tls)
Category: Cloud & infrastructure
Level: Intermediate
Tags: dns, http, tls, load-balancing, curl
Last reviewed: 2026-08-13
A successful HTTPS request depends on several independent systems. Test them separately so a DNS problem is not mistaken for an application problem.
## Resolve the name [#resolve-the-name]
```bash
getent ahosts api.example.com
resolvectl query api.example.com
dig api.example.com A
dig api.example.com AAAA
dig api.example.com CNAME
```
`getent` uses the host’s configured name-service path and is closest to what many applications see. `dig` shows DNS records directly and makes it easier to compare resolvers.
```bash
dig @1.1.1.1 api.example.com A
dig +trace api.example.com
```
Querying another resolver can reveal a caching or split-DNS difference. `+trace` follows delegation from the DNS root and may be blocked on restricted networks. Do not use either result as proof that the application itself uses the same resolver path.
DNS changes are not immediate everywhere. Resolvers and clients can retain the previous answer until its TTL expires. Before a planned cutover, lower the TTL far enough in advance, confirm the new value is being served, and keep the old endpoint available during the transition.
## Test the port [#test-the-port]
```bash
nc -vz api.example.com 443
```
A successful connection proves a TCP handshake completed. A refusal usually means the destination replied but nothing accepted that port. A timeout can indicate a drop, missing route, unavailable host, or asymmetric return path; it does not identify which one by itself.
## Inspect TLS and HTTP [#inspect-tls-and-http]
```bash
curl -vI https://api.example.com
curl -sS -o /dev/null \
-w 'code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\n' \
https://api.example.com/health
```
Verbose curl output shows the selected address, connection, TLS negotiation, certificate result, request headers, and response headers. It can also expose credentials or tokens, so remove sensitive values before sharing it.
Test a new backend before changing public DNS:
```bash
curl --resolve api.example.com:443:203.0.113.25 \
-I https://api.example.com/health
```
`--resolve` directs this curl request to the chosen address while preserving the hostname for TLS and HTTP. It does not alter system DNS.
For certificate details:
```bash
openssl s_client \
-connect api.example.com:443 \
-servername api.example.com \
-verify_return_error load balancer ── HTTP :3000 ──> application
```
The application should trust forwarded client headers only from known proxies that replace, rather than blindly append to, untrusted values. Keep a request ID across proxy and application logs so one failed request can be followed through both connections.
## References [#references]
* [DNS concepts and facilities](https://www.rfc-editor.org/rfc/rfc1034)
* [`dig` manual](https://bind9.readthedocs.io/en/stable/manpages.html#dig-dns-lookup-utility)
* [HTTP semantics](https://www.rfc-editor.org/rfc/rfc9110)
* [`curl` manual](https://curl.se/docs/manpage.html)
---
# Networking for Cloud Engineers (/docs/networking)
Category: Cloud & infrastructure
Level: Foundation
Tags: networking, cloud, dns, routing, troubleshooting
Last reviewed: 2026-08-13
Networking becomes easier when you treat a request as a sequence of decisions. A name resolves to an address, the client selects a route, firewalls allow or deny traffic, a process accepts a port, and an application returns a response.
```text
client
└─ DNS → public address
└─ route → firewall → load balancer
└─ service port → application
└─ response
```
## Learning path [#learning-path]
1 · Foundation
Addressing and routing
Read interface addresses, prefixes, gateways, cloud subnets, and route decisions.
2 · Intermediate
DNS, HTTP, and TLS
Follow a hostname through resolution, connection, certificate validation, and HTTP.
3 · Intermediate
Cloudflare
Operate proxied DNS, edge security, DDoS controls, and private application access.
4 · Intermediate
Network troubleshooting
Locate the failing layer with evidence instead of changing several systems at once.
## Working vocabulary [#working-vocabulary]
| Term | Practical meaning |
| ------------- | ------------------------------------------------------------------- |
| Address | Identifies an interface on an IP network |
| Prefix | Defines which addresses belong to a network, such as `/24` or `/64` |
| Route | Tells the host where to send traffic for a destination |
| Socket | A protocol, local address, and port used by a process |
| Firewall | Applies allow or deny policy to traffic |
| NAT | Rewrites addresses, commonly at an IPv4 network boundary |
| Load balancer | Accepts client traffic and selects a healthy backend |
## Read a Linux host without changing it [#read-a-linux-host-without-changing-it]
```bash
ip -brief link
ip -brief address
ip route
ip -6 route
resolvectl status
ss -lntup
```
These commands answer six useful questions: which interfaces are up, which addresses they own, where IPv4 and IPv6 traffic goes, which DNS resolver is active, and which processes are listening.
Describe the failure precisely
Record the source, destination, protocol, port, expected result, actual result, and timestamp. “The API is down” is vague. “Host A times out connecting to api.example.com:443, while DNS returns the expected address” identifies the next layer to inspect.
## Safety boundaries [#safety-boundaries]
* Keep the cloud provider console available before changing an interface, route, firewall, or SSH path.
* Capture the current configuration and prepare the exact rollback command first.
* Change one layer at a time, then repeat the same test.
* Treat packet captures as sensitive; they can contain internal addresses, hostnames, tokens, and unencrypted payloads.
## References [#references]
* [`ip-route` manual](https://man7.org/linux/man-pages/man8/ip-route.8.html)
* [`ss` manual](https://man7.org/linux/man-pages/man8/ss.8.html)
* [`resolvectl` manual](https://www.freedesktop.org/software/systemd/man/latest/resolvectl.html)
---
# Network Troubleshooting (/docs/networking/troubleshooting)
Category: Operations
Level: Intermediate
Tags: networking, troubleshooting, tcpdump, sockets, incidents
Last reviewed: 2026-08-13
Start from one failing source and one destination. Keep the test unchanged while you move through the layers.
## 1. Confirm the local state [#1-confirm-the-local-state]
```bash
date -Is
hostnamectl --static
ip -brief link
ip -brief address
ip route
resolvectl status
```
Record the timestamp and host. Check that the expected interface is up, has the expected address, and has a route for the destination.
## 2. Resolve DNS [#2-resolve-dns]
```bash
getent ahosts api.example.com
dig api.example.com A +short
dig api.example.com AAAA +short
```
Compare the answer with the intended load balancer or server. Test both address families when both are published.
## 3. Confirm the route [#3-confirm-the-route]
```bash
ip route get 203.0.113.25
tracepath 203.0.113.25
```
`ip route get` shows the local decision without sending traffic. `tracepath` can identify where replies stop and reveal path-MTU information, but missing hops are not proof of a failure because routers may suppress diagnostic responses.
## 4. Test the socket [#4-test-the-socket]
On the client:
```bash
nc -vz -w 5 api.example.com 443
```
On the server or backend:
```bash
sudo ss -lntp 'sport = :443 or sport = :3000'
sudo nft list ruleset
```
Confirm the service is bound to the intended address. A listener on `127.0.0.1:3000` is available only from the same host. A listener on `0.0.0.0:3000` accepts IPv4 traffic on every interface if policy permits it.
## 5. Test TLS and HTTP [#5-test-tls-and-http]
```bash
curl -vI --connect-timeout 5 https://api.example.com
curl -fsS http://127.0.0.1:3000/health
```
If the local health check succeeds but the public request fails, focus on the proxy, load balancer, certificate, and firewall. If both fail, inspect the application process and its logs first.
## 6. Correlate cloud policy [#6-correlate-cloud-policy]
Check the complete path in both directions:
* source subnet route and outbound policy;
* destination subnet route and inbound policy;
* network ACLs or equivalent stateless rules;
* security groups or equivalent stateful rules;
* load-balancer listener, target port, and health result; and
* host firewall and service binding.
Avoid temporarily allowing all traffic in production. It hides the real rule and creates a second incident risk.
## Capture only when needed [#capture-only-when-needed]
```bash
sudo tcpdump -ni any \
'host 203.0.113.25 and tcp port 443' \
-c 100 -w /tmp/api-443.pcap
```
Use the narrowest useful filter and a packet limit. Protect the capture as incident data and delete it through the team’s approved retention process after analysis.
## Write the finding [#write-the-finding]
Use a statement another engineer can verify:
```text
09:42 UTC from web-03:
- DNS returned 203.0.113.25 as expected.
- The kernel selected eth0 through 10.20.4.1.
- TCP 443 completed in 18 ms.
- TLS succeeded for api.example.com.
- The load balancer returned HTTP 502.
- The backend health check on 127.0.0.1:3000 was refused.
```
This points to the backend process or service configuration without changing DNS, routes, or public firewall rules.
## References [#references]
* [`ss` socket inspection](https://man7.org/linux/man-pages/man8/ss.8.html)
* [`ip-route` lookup](https://man7.org/linux/man-pages/man8/ip-route.8.html)
* [`curl` diagnostics](https://curl.se/docs/manpage.html)
---
# Act on Evidence (/docs/problem-solving/act-on-evidence)
Category: Engineering
Level: Foundation
Tags: experiments, decisions, delivery, feedback, learning
Last reviewed: 2026-08-13
Action turns thinking into evidence. The goal is not activity; it is learning whether a change improves the outcome without creating unacceptable harm.
## Design the smallest useful test [#design-the-smallest-useful-test]
Define before starting:
```text
Change: What will be different?
Scope: Who, what, and where can be affected?
Expected: Which signal should improve, and by how much?
Guardrail: Which signal must not get worse?
Time: When will we review the result?
Owner: Who decides to continue, change, or stop?
Recovery: How do we return to a known state?
```
Prefer a test that is narrow, reversible, observable, and quick enough to teach the team something.
## Match action to risk [#match-action-to-risk]
| Decision | Approach |
| ---------------------------- | ------------------------------------------------------------------- |
| Easy to reverse | Act with a small test and fast feedback |
| Expensive to reverse | Gather more evidence and review assumptions |
| High user or security impact | Add domain review, controls, and explicit approval |
| Urgent incident | Stabilize first; investigate without making several changes at once |
## Avoid two traps [#avoid-two-traps]
**Analysis paralysis:** waiting for certainty that cannot exist. Set a timebox and choose the next safe learning step.
**Action bias:** changing something because movement feels useful. If the team cannot name the expected signal, the action is not ready.
## Review what happened [#review-what-happened]
Compare the result with the prediction. Record:
* what changed and what did not;
* unexpected effects;
* whether the explanation still fits;
* the decision to keep, adjust, revert, or investigate; and
* the new question created by the result.
AI can draft a test plan or summarize results, but the engineer must verify inputs, measurements, and conclusions. A generated explanation does not replace observed behavior.
---
# Problem Solving (/docs/problem-solving)
Category: Engineering
Level: Foundation
Tags: problem-solving, critical-thinking, decisions, innovation, ai
Last reviewed: 2026-08-13
Problem solving is the work of reducing uncertainty until the next useful action is clear. Strong problem solvers do not rush to answers. They define the real gap, use evidence, test small changes, and learn from the result.
## The working loop [#the-working-loop]
```text
Observe → Define → Explain → Choose → Test → Learn
↑ │
└──────────────── repeat ──────────────────┘
```
1. **Observe reality.** What is happening, for whom, and how often?
2. **Define the gap.** What should happen instead?
3. **Explain carefully.** Which causes fit the evidence? What remains unknown?
4. **Choose a response.** Prefer the smallest useful and reversible step.
5. **Test the result.** Decide the success and failure signals before acting.
6. **Learn and repeat.** Keep, change, or stop based on what happened.
A slow deployment, failed login, or unhappy customer is a symptom. Find the affected outcome, conditions, and evidence before choosing a fix.
## Questions that improve thinking [#questions-that-improve-thinking]
* What outcome matters?
* What do we know, and how do we know it?
* What are we assuming?
* Who sees the problem differently?
* What is the smallest test that could change our mind?
* What new risk could our solution create?
* How will we know the problem is actually better?
## Problem solving with AI [#problem-solving-with-ai]
AI can summarize evidence, challenge a theory, generate options, and draft an experiment. It can also repeat a false assumption or create a convincing answer without enough context.
| Use AI to | Keep with the engineer |
| ------------------------------------------------- | ------------------------------------------- |
| Find missing questions and competing explanations | Define the real outcome and affected people |
| Compare options against stated constraints | Check facts, context, and consequences |
| Draft tests and failure scenarios | Choose the action and acceptable risk |
| Summarize verified learning | Own the decision and its result |
Ask AI for alternatives and disconfirming evidence, not only agreement. Verify important claims against the system, users, data, and primary sources.
## Learning path [#learning-path]
## References [#references]
* [Lean Enterprise Institute: A3 problem solving](https://www.lean.org/lexicon-terms/a3-report/)
* [DORA: State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/)
* [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
---
# Innovate Smarter (/docs/problem-solving/innovate-smarter)
Category: Engineering
Level: Foundation
Tags: innovation, experiments, creativity, product-thinking, ai
Last reviewed: 2026-08-13
Innovation is not novelty. It is a better outcome under real constraints. Start from a meaningful problem, create several options, and test the riskiest assumption before building the whole solution.
## Open, then narrow [#open-then-narrow]
First, expand the option space:
* remove a step instead of automating it;
* change the sequence, owner, or boundary;
* reuse a proven pattern from another domain;
* combine two simple ideas;
* design for the failure path first; and
* ask what becomes possible if one constraint changes.
Then compare options:
| Question | Why it matters |
| --------------------------------- | ------------------------------------------ |
| Does it improve the user outcome? | Prevents technology from becoming the goal |
| What must be true for it to work? | Exposes the riskiest assumption |
| How quickly can we learn? | Avoids a long build before feedback |
| Is it safe and reversible? | Limits the cost of being wrong |
| Can the team operate it? | Protects long-term value |
## Prototype the uncertainty [#prototype-the-uncertainty]
Do not prototype every feature. Prototype what the team knows least about: user value, technical feasibility, integration behavior, cost, security, or operational load.
A good prototype answers a question. Decide the question and evidence before building it. Stop when the answer is clear.
## Use AI without becoming average [#use-ai-without-becoming-average]
AI is useful for producing many starting points, combining patterns, and challenging a design. Because it often returns common patterns, accepting the first answer can make every solution look the same.
Ask for options with different trade-offs. Add real company context, user evidence, and constraints. Reject invented facts and generic features. The final idea should reflect what the team has learned—not only what the model has seen before.
## Make learning durable [#make-learning-durable]
When an experiment works, turn it into a maintained capability: simplify the implementation, add tests and observability, document the decision, name an owner, and remove the temporary path.
When it fails, preserve the useful learning. A clear invalidated assumption can save the next team from repeating the same expensive idea.
---
# Think Clearly (/docs/problem-solving/think-clearly)
Category: Engineering
Level: Foundation
Tags: critical-thinking, problem-framing, root-cause, evidence
Last reviewed: 2026-08-13
Clear thinking starts by describing reality without hiding a conclusion inside the problem statement.
## Write the gap [#write-the-gap]
```text
For [affected user or system],
[observed behavior] happens under [conditions].
We expected [target behavior].
The impact is [measured effect].
We will know it improved when [signal changes].
```
Weak: “The database is too small.” This assumes the answer.
Better: “Checkout requests exceed the latency target during the 18:00 traffic peak; database connection wait accounts for most of the delay.”
## Separate what you know [#separate-what-you-know]
| Type | Example |
| ---------- | ----------------------------------------------------------- |
| Fact | The trace shows 800 ms waiting for a database connection |
| Assumption | A larger database will reduce the wait |
| Unknown | Whether the pool is exhausted by load or leaked connections |
| Constraint | Checkout must remain available during investigation |
Facts need a source. Assumptions need a test. Unknowns need a next question.
## Find causes, not blame [#find-causes-not-blame]
Ask “what conditions allowed this?” before “who changed it?” Trace the system from the visible symptom toward inputs, dependencies, state, recent changes, and controls.
A cause is useful when changing it should change the outcome. Test that relationship. A timeline match or confident story is not enough.
## Use AI as a challenger [#use-ai-as-a-challenger]
Give AI the sanitized problem statement and evidence. Ask:
* Which assumptions are hidden here?
* What other explanations fit the same facts?
* What evidence would disprove each explanation?
* Which question should we answer first?
Do not ask AI to name the root cause from partial context. Its job is to widen the inquiry; evidence narrows it.
## Ready to act [#ready-to-act]
Move forward when the team can state the outcome, evidence, key assumptions, affected people, constraints, and the next test. Perfect certainty is not required. A clear learning step is.
---
# Detection and Response (/docs/security/detection-and-response)
Category: Security operations
Level: Intermediate
Tags: monitoring, logging, detection, incident-response, forensics
Last reviewed: 2026-08-13
More logs do not automatically create better detection. Collect events that answer a question, keep enough context to investigate them, and route each alert to someone with authority to act.
## Build useful telemetry [#build-useful-telemetry]
| Source | Events worth keeping | Questions it answers |
| ------------------- | ---------------------------------------------------------------- | -------------------------------------------------------- |
| Identity | Sign-ins, failures, MFA changes, privilege grants | Who accessed what, from where, and with which privilege? |
| Linux host | SSH, `sudo`, service, package, process, file-integrity events | What changed on the host? |
| Network | Flow logs, firewall decisions, DNS, load-balancer and proxy logs | Which systems communicated and what was blocked? |
| Application | Authentication, authorization, admin actions, errors | Which user or service initiated the action? |
| Cloud control plane | API calls, policy changes, key use, snapshot actions | Who changed infrastructure or security controls? |
Send security-relevant logs off the server quickly. Synchronize time, restrict log deletion, document retention, and avoid recording secrets, session tokens, full credentials, or unnecessary personal data.
## Inspect one Linux host [#inspect-one-linux-host]
```bash
date -Is
uptime
who
last -Fai | head -30
sudo journalctl --since '2 hours ago' -p warning
sudo journalctl -u ssh --since '2 hours ago'
sudo journalctl _COMM=sudo --since '2 hours ago'
sudo ss -tpna
ps -eo user,pid,ppid,lstart,cmd --sort=-lstart | head -40
```
These commands support triage, not a verdict. Compare results with deployment records, configuration management, expected operators, and the service baseline.
## Write alerts as decisions [#write-alerts-as-decisions]
An actionable alert states:
* what changed and which detection produced the signal;
* affected account, host, service, and environment;
* first and last observed time, count, and relevant baseline;
* evidence links that do not expose secrets;
* likely impact and confidence;
* the first safe check, escalation owner, and containment option.
Examples of useful signals include a new public listener, interactive login by a service account, security logging disabled, a burst of failed logins followed by success, unexpected privilege escalation, or outbound traffic to a destination never used by the service.
## Respond in a controlled order [#respond-in-a-controlled-order]
Declare and assign
Name the incident lead, communications owner, operations lead, and evidence owner. Record decisions and times in one shared timeline.
Confirm scope
Identify affected identities, hosts, data, regions, and dependencies. Separate confirmed facts from working hypotheses.
Contain safely
Use the cloud or network control plane to isolate affected systems. Avoid powering off a host when volatile evidence may matter unless safety or ongoing damage requires it.
Remove access
From a known-clean system, revoke sessions, rotate exposed credentials, remove persistence, and close the initial access path.
Recover trust
Rebuild from approved artifacts, restore clean data, validate controls, and monitor the recovered service for recurrence.
## Preserve evidence [#preserve-evidence]
Record who collected each artifact, when, from where, how its integrity was checked, and every transfer. Follow company policy and legal guidance for packet captures, disk snapshots, memory images, personal data, and communications. Do not run unreviewed cleanup commands that destroy timestamps or logs.
## Improve after the incident [#improve-after-the-incident]
A useful review explains impact, timeline, contributing conditions, why safeguards did not prevent or detect the event sooner, and what will change. Give every action an owner, priority, due date, and verification method. Focus on system conditions and decision context rather than blame.
## References [#references]
* [NIST SP 800-61 Rev. 3: Incident Response](https://csrc.nist.gov/pubs/sp/800/61/r3/final)
* [NIST Cybersecurity Framework 2.0](https://www.nist.gov/cyberframework)
* [Google SRE: Postmortem Culture](https://sre.google/sre-book/postmortem-culture/)
---
# Honeypots (/docs/security/honeypots)
Category: Security operations
Level: Intermediate
Tags: honeypot, deception, monitoring, opencanary, detection
Last reviewed: 2026-08-13
A honeypot is a decoy, not a counterattack. It exposes a service that legitimate users should not need, so interaction can create a high-signal alert. It does not replace patching, endpoint monitoring, segmentation, or incident response.
## Safe design rules [#safe-design-rules]
* Put the decoy in a dedicated account, project, network segment, or tightly controlled subnet.
* Give it no production credentials, customer data, mounted secrets, trusted keys, or administrative path.
* Deny or tightly restrict outbound traffic so a compromised decoy cannot attack other systems.
* Send alerts and logs to a separate trusted system that the decoy cannot modify.
* Use an intentionally fictional server identity. Do not copy real documents or personal data.
* Obtain security, legal, privacy, and network-owner approval before exposing or recording traffic.
* Define how to isolate, preserve, rebuild, and retire the decoy.
Never use a honeypot to run code on a visitor’s system, steal data, damage infrastructure, or pursue an attacker. Source addresses can be spoofed or belong to compromised third parties. Use the signal to protect your own environment and report through approved channels.
## Quick OpenCanary lab [#quick-opencanary-lab]
OpenCanary can emulate services and send an alert when they are used. Start in a disposable lab network—not on a production host.
```bash
python3 -m venv /opt/opencanary/venv
sudo /opt/opencanary/venv/bin/pip install --upgrade pip opencanary
sudo /opt/opencanary/venv/bin/opencanaryd --copyconfig
```
Edit the generated configuration to set a unique `device.node_id`, enable only the decoy services you intend to monitor, and send alerts to a protected destination. Validate the JSON and start in the foreground first:
```bash
sudo jq . /root/.opencanary.conf
sudo /opt/opencanary/venv/bin/opencanaryd --dev
```
Paths vary by the service account and packaging method. Follow the current [OpenCanary getting-started guide](https://docs.opencanary.org/en/latest/starting/opencanary.html), pin an approved version, and manage the final process with a reviewed system service or container definition.
## Network policy [#network-policy]
The safest useful pattern is:
```text
untrusted or monitored network
│
▼
decoy subnet ── alerts ──▶ protected log collector
│
└── outbound: deny by default
production networks: no route or explicit deny
```
If the decoy needs DNS, time synchronization, updates, or an alert webhook, allow only those named destinations through a controlled egress path. Do not grant broad internet access.
## Test before trusting it [#test-before-trusting-it]
From an authorized test host:
1. Connect once to an enabled decoy service.
2. Confirm the expected event reaches the central collector.
3. Confirm the alert identifies the decoy, source, destination, service, and time.
4. Verify the on-call route and runbook link.
5. Confirm the decoy cannot reach production or arbitrary internet destinations.
6. Rebuild the decoy from its approved definition.
## Triage an alert [#triage-an-alert]
Treat interaction as suspicious, not automatically malicious. Check whether the source is an approved scanner, a configuration mistake, an internal host, or an external address. Then correlate identity, DNS, flow, firewall, and endpoint events around the same time.
Escalate when the source is internal, multiple decoys are touched, the behavior follows a deliberate sequence, production systems show related activity, or the decoy attempts unexpected outbound communication.
## Operate it like a security sensor [#operate-it-like-a-security-sensor]
* Keep ownership, patching, alert testing, and expiry dates in the asset inventory.
* Measure alert-delivery failures and sensor silence.
* Rotate the decoy personality when it no longer represents a useful detection opportunity.
* Rebuild after confirmed compromise; do not treat the decoy as a trusted forensic workstation.
* Retire unused decoys so they do not become forgotten internet-facing assets.
## References [#references]
* [OpenCanary documentation](https://docs.opencanary.org/en/latest/)
* [NIST SP 800-61 Rev. 3: Incident Response](https://csrc.nist.gov/pubs/sp/800/61/r3/final)
---
# Security for Cloud Operations (/docs/security)
Category: Security
Level: Foundation
Tags: security, linux, cloud, hardening, incident-response
Last reviewed: 2026-08-13
Security is a continuing operating practice. No firewall, agent, or scanner can guarantee that a server will not be compromised. A useful program reduces exposure, makes abnormal behavior visible, limits how far an attacker can move, and prepares the team to recover.
## Work across the full cycle [#work-across-the-full-cycle]
| Area | Question | Useful evidence |
| -------- | ------------------------------------------ | -------------------------------------------------------- |
| Govern | Who owns the risk and the decision? | Service owner, policy, escalation path |
| Identify | What exists and what matters most? | Asset inventory, data classification, dependency map |
| Protect | Which controls reduce likely attack paths? | Access policy, patch state, segmentation, backups |
| Detect | How will the team notice a change? | Central logs, endpoint events, network telemetry, alerts |
| Respond | Who can contain the incident safely? | Runbook, roles, communications plan, preserved evidence |
| Recover | Can the service return to a trusted state? | Tested restore, clean images, rotated credentials |
This follows the six functions in the [NIST Cybersecurity Framework 2.0](https://www.nist.gov/cyberframework). Use the framework to organize decisions, then choose controls that fit the service and its risk.
## Learning path [#learning-path]
## Start with ownership [#start-with-ownership]
For every internet-facing service, record:
* a named service owner and a security contact;
* the public hosts, ports, domains, data, and upstream dependencies;
* the maximum acceptable outage and data loss;
* where logs and backups are stored, and who can access them; and
* who may isolate a host, revoke credentials, or fail traffic over during an incident.
Controls without an owner quietly decay. Ownership turns a checklist into an operating system for decisions.
Do not break into, damage, or disrupt systems believed to belong to an attacker. They may be compromised third-party systems, and retaliation creates legal, safety, and evidence risks. Contain your environment, block malicious traffic, preserve evidence, and use the approved reporting or law-enforcement path.
## When you suspect compromise [#when-you-suspect-compromise]
1. Open an incident channel and assign an incident lead.
2. Record the time, affected assets, symptoms, and source of the alert.
3. Isolate affected systems through the cloud or network control plane when possible.
4. Preserve relevant logs, volatile evidence, and disk snapshots according to policy.
5. Revoke exposed sessions and credentials from a known-clean system.
6. Rebuild from trusted artifacts; do not return an unexplained host to service.
7. Validate recovery, monitor closely, and track corrective work to completion.
The current [NIST incident-response guidance](https://csrc.nist.gov/pubs/sp/800/61/r3/final) treats preparation, detection, response, recovery, and improvement as connected risk-management work—not a process that starts only after an alert.
---
# Server Protection (/docs/security/server-protection)
Category: Security
Level: Foundation
Tags: hardening, linux, patching, access-control, backups
Last reviewed: 2026-08-13
Hardening begins with knowing why a server exists. Keep only the packages, listeners, identities, and data needed for that purpose. A smaller system is easier to patch, observe, and rebuild.
## Establish the baseline [#establish-the-baseline]
Capture the state before changing it:
```bash
date -Is
hostnamectl
uname -r
ip -brief address
ip route
sudo ss -lntup
systemctl --failed
systemctl list-unit-files --state=enabled
```
Compare every listening socket and enabled service with an approved requirement. Investigate unknown items before removing them; a port may belong to monitoring, cluster coordination, or a local-only application.
## Patch by server family [#patch-by-server-family]
```bash
sudo apt update
apt list --upgradable
sudo apt upgrade
test -f /var/run/reboot-required && cat /var/run/reboot-required
```
Use unattended security updates only after the team has defined maintenance windows, restart behavior, health checks, and rollback. Review held packages with `apt-mark showhold`.
```bash
sudo dnf check-update
sudo dnf updateinfo list --security
sudo dnf upgrade --security
sudo dnf needs-restarting -r
```
On systems using a managed repository or lifecycle service, confirm that the host receives the intended release stream before applying changes.
Prioritize internet-facing assets and vulnerabilities known to be exploited. CISA maintains the [Known Exploited Vulnerabilities Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) as an input to risk-based remediation; it is not a substitute for a complete vulnerability program.
## Control administrative access [#control-administrative-access]
* Require individual identities. Do not share administrator accounts.
* Prefer short-lived access through an identity-aware gateway, VPN, or session manager.
* Disable direct root login and password authentication only after key-based access and a recovery path have been tested.
* Grant the smallest practical `sudo` scope and review it regularly.
* Remove access promptly when a role changes or a person leaves.
```bash
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|pubkeyauthentication|maxauthtries) '
sudo visudo -c
sudo last -ai | head
sudo journalctl -u ssh --since '24 hours ago'
```
`sshd -T` shows effective settings after includes and defaults. Make one access change at a time and keep an existing tested session open until a second session succeeds.
## Restrict the network path [#restrict-the-network-path]
Use layers with distinct jobs:
1. A cloud firewall or security group allows only required sources and ports.
2. A load balancer or reverse proxy terminates public traffic where appropriate.
3. The host firewall mirrors the intended exposure.
4. The application binds to the narrowest useful address.
5. Databases and management services stay on private networks.
```bash
sudo ss -lntup
sudo nft list ruleset
curl -fsS http://127.0.0.1:3000/health
```
Do not use a temporary allow-from-anywhere rule as a troubleshooting shortcut. Test one path from a known source and inspect each control in order.
## Protect data and recovery [#protect-data-and-recovery]
* Encrypt disks, snapshots, object storage, and backup traffic with managed keys.
* Keep at least one backup copy outside the credentials and administration path of production.
* Protect deletion and retention settings with stronger authorization than routine writes.
* Restore into an isolated environment on a schedule and record recovery time and data loss.
* Store infrastructure definitions and approved images so a compromised host can be replaced.
Definition of done
The approved listeners are the only listeners, administrative access is attributable, critical patches meet the service deadline, logs leave the host, and a recent restore test has evidence.
## References [#references]
* [CISA Cross-Sector Cybersecurity Performance Goals](https://www.cisa.gov/cross-sector-cybersecurity-performance-goals)
* [CISA Known Exploited Vulnerabilities Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
* [NIST Cybersecurity Framework 2.0](https://www.nist.gov/cyberframework)
---
# Threat Resilience (/docs/security/threat-resilience)
Category: Security operations
Level: Advanced
Tags: ddos, malware, ransomware, zero-day, vulnerability-management
Last reviewed: 2026-08-13
Threat names describe different failure modes. Use a separate detection and response plan for each one, while sharing the same asset inventory, identity controls, telemetry, and recovery process.
## DDoS [#ddos]
DDoS protection must exist upstream of the saturated resource. A host firewall cannot recover bandwidth that is already exhausted.
**Prepare**
* Put public HTTP services behind a provider with network and application-layer DDoS protection.
* Define which endpoints are essential and which can be degraded, cached, queued, or disabled.
* Set bounded timeouts, connection limits, request-size limits, and rate policies at the edge.
* Protect the origin so traffic cannot bypass the edge service.
* Pre-arrange provider escalation and test it during an exercise.
**Monitor**
* requests, connections, packets, and bits per second;
* source and destination distribution, protocols, and response codes;
* edge blocks and challenges, origin saturation, queue depth, and dependency health; and
* user-visible availability from more than one network.
**Respond**
Confirm the constrained layer, contact the upstream provider early, apply narrow mitigations, preserve legitimate access, and communicate service impact. Scaling may help with application load but is not a complete defense against volumetric attacks. See the joint [CISA, FBI, and MS-ISAC DDoS guidance](https://www.cisa.gov/resources-tools/resources/understanding-and-responding-distributed-denial-service-attacks).
## Malware [#malware]
**Prepare:** minimize software, restrict execution and administrative access, scan incoming artifacts, protect build systems, centralize endpoint telemetry, and segment services by trust.
**Watch for:** an unexpected process tree, execution from temporary or writable directories, disabled security tooling, new persistence, mass file changes, unusual DNS, or unexplained outbound connections.
**Respond:** isolate the host, preserve evidence, block confirmed indicators, rotate exposed credentials, determine the entry path, and rebuild from trusted artifacts. Deleting the first suspicious file does not prove the system is clean.
## Ransomware and destructive activity [#ransomware-and-destructive-activity]
Ransomware can combine encryption, deletion, credential theft, and data extortion.
* Separate routine production administration from backup administration.
* Keep protected, versioned, and offline or logically isolated recovery copies.
* Alert on bulk deletion, backup-policy changes, snapshot deletion, unusual encryption activity, and security-control changes.
* Rehearse restoring identity, configuration, data, and dependencies in the correct order.
* Decide in advance who coordinates legal, privacy, insurance, law enforcement, customer, and executive communications.
Do not assume that paying will restore systems or prevent disclosure. Follow the company’s legal and incident leadership process. The [CISA StopRansomware guide](https://www.cisa.gov/stopransomware/ransomware-guide) and [NIST ransomware profile](https://csrc.nist.gov/pubs/ir/8374/r1/final) cover prevention, response, and recovery as one program.
## Newly disclosed and zero-day vulnerabilities [#newly-disclosed-and-zero-day-vulnerabilities]
A zero-day has no guaranteed preventive control. The objective is to know exposure quickly, apply trustworthy mitigations, detect exploitation, and restore a patched or replaced service.
Verify the advisory
Use the vendor, CISA, or another authoritative source. Record affected versions, exploitation status, indicators, mitigations, and uncertainty.
Find exposure
Query the asset and software inventory. Include appliances, containers, images, libraries, CI runners, and dormant internet-facing systems.
Reduce reachability
Disable an affected feature, restrict the route, remove public exposure, add a vendor-approved rule, or stop the service when business impact permits.
Hunt and observe
Search historical logs for published indicators and behavior. Increase targeted telemetry without overwhelming the response team.
Patch and validate
Test the vendor fix, deploy by exposure and impact, verify the installed version, and remove temporary controls only after validation.
Prioritize vulnerabilities with evidence of active exploitation, but continue addressing serious weaknesses before they reach the [CISA Known Exploited Vulnerabilities Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog).
## A useful operations dashboard [#a-useful-operations-dashboard]
Show service health and security context together:
* public availability, latency, errors, saturation, and dependency health;
* edge traffic, blocks, connection patterns, and origin reachability;
* privileged access, identity-policy changes, and failed-to-successful logins;
* endpoint health, suspicious processes, file-integrity changes, and sensor gaps;
* patch age, internet exposure, known exploited vulnerabilities, and expiring exceptions; and
* last successful backup, last restore test, recovery time, and unresolved recovery gaps.
Every panel should have an owner, a decision it supports, and a runbook link. A dashboard nobody uses during an incident is decoration.
---
# DNS and Reverse Proxy (/docs/networking/cloudflare/dns-and-proxy)
Category: Networking
Level: Intermediate
Tags: cloudflare, dns, proxy, tls, origin
Last reviewed: 2026-08-13
The proxy status on a DNS record determines whether eligible HTTP or HTTPS traffic goes through Cloudflare or directly to the record target.
## Choose the record mode [#choose-the-record-mode]
| Mode | DNS answer | Appropriate use |
| -------- | ---------------------------------- | -------------------------------------------------------------------------------- |
| Proxied | Cloudflare anycast addresses | Public HTTP and HTTPS applications |
| DNS only | The origin address or CNAME target | Mail, domain verification, unsupported protocols, or a deliberate direct service |
Only eligible `A`, `AAAA`, and `CNAME` records can be proxied. `MX`, `TXT`, and other record types remain DNS-only. Non-HTTP services and unsupported ports need another product or architecture; changing the cloud icon does not turn every protocol into proxied traffic.
## Prepare a DNS cutover [#prepare-a-dns-cutover]
1. Export and review the current zone.
2. Remove stale records and identify every record that reveals an origin address.
3. Lower TTLs at the current provider early enough for caches to expire.
4. Recreate records and mark only eligible web hostnames as proxied.
5. Validate mail, verification, certificate, and application records before changing nameservers.
6. If DNSSEC is active, follow the provider migration sequence; a stale DS record can make the zone fail validation.
7. After activation, enable DNSSEC and publish the new DS record at the registrar.
```bash
dig example.com NS +short
dig app.example.com A +short
dig app.example.com AAAA +short
dig example.com MX +short
dig example.com DS +short
```
A proxied hostname should normally return Cloudflare addresses rather than the configured origin. Compare results from more than one resolver during a cutover.
## Encrypt both connections [#encrypt-both-connections]
Cloudflare terminates the client connection and creates another connection to the origin. Use `Full (strict)` so the origin presents a valid, unexpired certificate matching the hostname.
```bash
curl -sSvo /dev/null https://app.example.com
openssl s_client -connect origin.example.internal:443 \
-servername app.example.com
1 · Foundation
DNS and reverse proxy
Choose proxied or DNS-only records, validate TLS, and prevent direct origin access.
2 · Intermediate
WAF and DDoS
Deploy managed rules, endpoint-aware rate limits, and a practical response workflow.
3 · Intermediate
Tunnel and Access
Publish services through outbound connections and protect private applications by identity.
## Know which control is active [#know-which-control-is-active]
| Capability | What it does | What it does not prove |
| ----------------- | ---------------------------------------------------------- | ---------------------------------------------- |
| Authoritative DNS | Answers queries for the zone | Traffic passes through Cloudflare |
| Proxied record | Routes eligible web traffic through Cloudflare | The origin rejects direct traffic |
| WAF | Evaluates matching HTTP requests | The application has no vulnerabilities |
| DDoS protection | Detects and mitigates attack traffic at the edge | The origin cannot be reached directly |
| Tunnel | Connects Cloudflare to a service over outbound connections | A user is authorized to access it |
| Access | Applies identity and device-aware policy | The application’s own authorization is correct |
## Recommended baseline [#recommended-baseline]
1. Proxy eligible public web hostnames; keep mail, verification, and unsupported services DNS-only.
2. Use `Full (strict)` TLS with a valid certificate at the origin.
3. Restrict the origin to Cloudflare traffic, or remove public ingress by using Tunnel.
4. Enable an appropriate managed WAF ruleset and review its events before adding exceptions.
5. Add rate limits for login, authentication, expensive search, upload, and API paths based on normal traffic.
6. Put internal applications behind Access policies or private-network routes; do not publish them first and add identity later.
7. Alert on origin errors, security events, tunnel health, DNS changes, and policy changes.
8. Keep a tested path to the origin for incident response that does not bypass normal authorization.
Cloudflare reduces exposure and absorbs traffic before it reaches your service. Keep application authentication, authorization, input validation, patching, logging, backups, and incident response in place.
## References [#references]
* [How Cloudflare DNS and reverse proxy work](https://developers.cloudflare.com/fundamentals/concepts/how-cloudflare-works/)
* [Protect an origin server](https://developers.cloudflare.com/fundamentals/security/protect-your-origin-server/)
* [Cloudflare Tunnel](https://developers.cloudflare.com/tunnel/)
---
# Tunnel and Access (/docs/networking/cloudflare/tunnel-and-access)
Category: Networking
Level: Intermediate
Tags: cloudflare, tunnel, zero-trust, access, private-network
Last reviewed: 2026-08-13
Cloudflare Tunnel creates outbound connections from `cloudflared` to Cloudflare, so a service can be reached without opening a public inbound port. Cloudflare Access is the authorization layer that decides who or what may reach a protected application.
## Choose the route type [#choose-the-route-type]
| Need | Route | Client requirement |
| ----------------------------------------- | ------------------------------------- | ------------------------------------------ |
| Public web application through Cloudflare | Published application hostname | Browser or normal HTTP client |
| Internal web application by identity | Published hostname plus Access policy | Browser and configured identity provider |
| Private IP or non-HTTP network access | Private network route | Cloudflare One Client or connected network |
| Machine-to-machine application | Access service authentication | Protected service credential flow |
A published hostname without an Access application may be public. Configure the access decision before announcing or depending on the hostname.
## Build a public application Tunnel [#build-a-public-application-tunnel]
For most production use cases, Cloudflare recommends a remotely managed tunnel through the dashboard, API, or Terraform. The high-level flow is:
1. Create the tunnel and store its token through the approved secret-management path.
2. Install `cloudflared` on a host that can reach the application locally.
3. Map the public hostname to a narrow local service such as `http://127.0.0.1:8080`.
4. Add a final catch-all rule that returns `404` for unmatched hostnames when using local ingress configuration.
5. Run at least two replicas on separate failure domains when the application requires high availability.
6. Allow required outbound Tunnel traffic and deny public inbound traffic to the origin.
```bash
cloudflared version
cloudflared tunnel list
cloudflared tunnel info app-prod
```
Do not paste a tunnel token, account certificate, or credentials file into source control, tickets, documentation, shell transcripts, or chat. Treat them as privileged secrets and rotate them after exposure.
## Define Access before use [#define-access-before-use]
Cloudflare Access is deny-by-default for requests that do not match an Allow policy. Build policy from specific identity and device requirements:
* include the smallest team or group that needs the application;
* require MFA and device posture for sensitive administrative tools;
* use short sessions for higher-risk applications;
* use Service Auth for approved non-user clients;
* avoid Bypass unless the traffic truly must skip Access controls; and
* test an unauthorized user, an authorized user, and a failed device posture before release.
Cloudflare also offers an account setting that requires Access protection for hostnames, helping prevent a new internal hostname from being exposed before its Access application exists. Evaluate it carefully against intentionally public hostnames before enabling it.
## Connect a private network [#connect-a-private-network]
For private IP routes, install `cloudflared` in the network, advertise only the required CIDR ranges, enroll user devices in the Zero Trust organization, and apply Gateway network policy where needed.
Do not advertise an entire cloud network when users need one subnet or service. Keep production, management, and shared-service routes distinct so policy and incident containment remain understandable.
## Operate the connector [#operate-the-connector]
Monitor:
* connector and replica health;
* application reachability from `cloudflared`;
* Access allow and deny events;
* identity-provider and device-posture failures;
* tunnel configuration and route changes; and
* origin latency and errors after traffic enters the tunnel.
A Tunnel can remain connected while the local application is unhealthy. Test the complete path, not only connector status.
## Recovery checks [#recovery-checks]
```bash
cloudflared tunnel list
cloudflared tunnel info app-prod
curl -fsS http://127.0.0.1:8080/health
curl -sSI https://app.example.com
```
Confirm DNS still points to the intended tunnel, healthy replicas exist in separate failure domains, Access applies the expected decision, and the origin service is reachable locally.
## References [#references]
* [Cloudflare Tunnel](https://developers.cloudflare.com/tunnel/)
* [Tunnel routing](https://developers.cloudflare.com/tunnel/routing/)
* [Tunnel configuration and replicas](https://developers.cloudflare.com/tunnel/configuration/)
* [Cloudflare Access policies](https://developers.cloudflare.com/cloudflare-one/access-controls/policies/)
* [Private network routes](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/private-net/)
---
# WAF and DDoS (/docs/networking/cloudflare/waf-and-ddos)
Category: Web security
Level: Intermediate
Tags: cloudflare, waf, ddos, rate-limiting, monitoring
Last reviewed: 2026-08-13
Cloudflare’s edge can filter malicious HTTP requests and absorb attack traffic before it reaches the application. Effective protection still depends on knowing normal traffic, preventing origin bypass, and giving responders useful signals.
## Layer the controls [#layer-the-controls]
| Layer | Primary job | Useful signal |
| ------------------ | ---------------------------------------------------- | ------------------------------------------------ |
| DDoS managed rules | Detect and mitigate network or HTTP flood patterns | Mitigated traffic, protocol, rate, origin health |
| Managed WAF rules | Detect common exploit patterns | Rule, path, action, false-positive rate |
| Custom rules | Express application-specific allow or block policy | Match count and business impact |
| Rate limiting | Bound repeated use of selected endpoints | Requests per key, action, affected users |
| Cache | Serve repeatable content without reaching the origin | Hit ratio, origin requests, stale behavior |
| Origin restriction | Stop traffic that bypasses Cloudflare | Direct connection attempts |
## Deploy managed rules deliberately [#deploy-managed-rules-deliberately]
1. Inventory frameworks, APIs, upload paths, authentication routes, and expected automation.
2. Enable the Cloudflare managed ruleset with its recommended defaults for the applicable plan.
3. Review Security Events and application errors during a representative traffic window.
4. Tune by rule, tag, path, or verified client when a legitimate request is affected.
5. Give every exception an owner, reason, narrow scope, and review date.
Do not enable every disabled rule without testing. Cloudflare notes that some rules are disabled to balance coverage and false positives. Likewise, do not create a broad skip rule to make one request work.
## Rate-limit behavior, not the whole site [#rate-limit-behavior-not-the-whole-site]
Start with endpoints where repeated requests have a clear cost or abuse pattern:
* login, password reset, and verification;
* expensive search or report generation;
* uploads and data exports;
* API routes with per-client quotas; and
* cache-bypass paths that reach costly dependencies.
Choose the counting key and threshold from observed traffic. Test the response your client receives, including `429` handling and retry behavior. A single IP can represent a shared corporate network, while an attacker may distribute requests across many IPs.
## Prepare for DDoS before the alert [#prepare-for-ddos-before-the-alert]
* Keep DDoS managed rules at the recommended sensitivity and mitigation action unless a reviewed exception requires otherwise.
* Restrict the origin to Cloudflare or use Tunnel.
* Cache safe content and prevent randomized query strings from defeating the intended cache policy.
* Bound origin connection, request, and application timeouts.
* Define which features can be degraded or temporarily disabled.
* Alert on edge traffic, mitigations, origin saturation, `5xx` responses, queue depth, and user-visible availability.
* Record the provider escalation path and practice it.
Cloudflare automatically mitigates large attacks, but attacks can still affect an application through origin exposure, expensive dynamic paths, dependencies, or rules that do not match the application’s behavior.
## Incident workflow [#incident-workflow]
1. Confirm whether the constrained resource is the edge, origin network, application, or dependency.
2. Compare edge request volume with origin request volume and user-visible health.
3. Identify the paths, methods, source distribution, cache status, and response codes driving impact.
4. Apply the narrowest effective managed, custom, rate, or cache control.
5. Watch legitimate success rates while mitigation is active.
6. Preserve event data and document every temporary rule.
7. Remove or convert emergency rules after review; do not leave unexplained blocks in place.
## References [#references]
* [Cloudflare managed WAF ruleset](https://developers.cloudflare.com/waf/managed-rules/reference/cloudflare-managed-ruleset/)
* [Rate limiting rules](https://developers.cloudflare.com/waf/rate-limiting-rules/)
* [Proactive DDoS defense](https://developers.cloudflare.com/ddos-protection/best-practices/proactive-defense/)
* [How Cloudflare DDoS protection works](https://developers.cloudflare.com/ddos-protection/about/how-ddos-protection-works/)