Services and networking
Run applications with systemd and trace requests across DNS, sockets, firewalls, and HTTP.
Most production failures become tractable when you can answer two questions: what process should be running? and how should traffic reach it?
Inspect processes and resources
Prefer SIGTERM and wait for graceful shutdown. Use SIGKILL only when the process cannot respond; it prevents cleanup and can leave state inconsistent.
Create a systemd service
Keep secrets in a root-owned file such as /etc/cubis-api.env with mode 600; do not store them in the unit or repository.
After changing application code, restart the service. After changing only the unit, run daemon-reload first.
Understand listening sockets
Binding to 127.0.0.1:3000 means only local processes can connect. Binding to 0.0.0.0:3000 exposes the socket on every IPv4 interface if the firewall allows it. An application behind Nginx should normally bind to loopback.
Trace the request path
Follow the layers in order instead of guessing:
Resolve DNS
dig +short app.example.com A should return the expected public IP. Also inspect AAAA if IPv6 is published.
Reach the host
nc -vz app.example.com 443 checks TCP reachability. A timeout suggests routing or firewall; refusal means the host answered but nothing accepted the port.
Negotiate TLS and HTTP
curl -vI https://app.example.com exposes DNS, connection, certificate, protocol, redirect, and response headers.
Reach the application locally
On the server, curl -v http://127.0.0.1:3000/health. If local works but public fails, focus on Nginx, TLS, or the firewall.
Correlate logs
Read Nginx access/error logs and the service journal for the same timestamp or request ID.
DNS and route tools
| Command | Use |
|---|---|
dig +short name A | Resolve IPv4 records |
resolvectl query name | Query through the host resolver |
ip -brief address | Show interface addresses |
ip route | Show routing decisions and default gateway |
tracepath host | Discover path and MTU issues |
curl -w '%{http_code} %{time_total}\n' | Measure HTTP result and total time |
Diagnostic habit State the failing layer: “DNS resolves, TCP 443 connects, TLS succeeds, Nginx returns 502, and the local health check refuses port 3000.” That sentence is far more actionable than “the server is down.”