Network Troubleshooting

Start from one failing source and one destination. Keep the test unchanged while you move through the layers.

1. Confirm the local state

date -Is
hostnamectl --static
ip -brief link
ip -brief address
ip route
resolvectl status

Record the timestamp and host. Check that the expected interface is up, has the expected address, and has a route for the destination.

2. Resolve DNS

getent ahosts api.example.com
dig api.example.com A +short
dig api.example.com AAAA +short

Compare the answer with the intended load balancer or server. Test both address families when both are published.

3. Confirm the route

ip route get 203.0.113.25
tracepath 203.0.113.25

ip route get shows the local decision without sending traffic. tracepath can identify where replies stop and reveal path-MTU information, but missing hops are not proof of a failure because routers may suppress diagnostic responses.

4. Test the socket

On the client:

nc -vz -w 5 api.example.com 443

On the server or backend:

sudo ss -lntp 'sport = :443 or sport = :3000'
sudo nft list ruleset

Confirm the service is bound to the intended address. A listener on 127.0.0.1:3000 is available only from the same host. A listener on 0.0.0.0:3000 accepts IPv4 traffic on every interface if policy permits it.

5. Test TLS and HTTP

curl -vI --connect-timeout 5 https://api.example.com
curl -fsS http://127.0.0.1:3000/health

If the local health check succeeds but the public request fails, focus on the proxy, load balancer, certificate, and firewall. If both fail, inspect the application process and its logs first.

6. Correlate cloud policy

Check the complete path in both directions:

• source subnet route and outbound policy;
• destination subnet route and inbound policy;
• network ACLs or equivalent stateless rules;
• security groups or equivalent stateful rules;
• load-balancer listener, target port, and health result; and
• host firewall and service binding.

Avoid temporarily allowing all traffic in production. It hides the real rule and creates a second incident risk.

Capture only when needed

sudo tcpdump -ni any \
  'host 203.0.113.25 and tcp port 443' \
  -c 100 -w /tmp/api-443.pcap

Use the narrowest useful filter and a packet limit. Protect the capture as incident data and delete it through the team’s approved retention process after analysis.

Write the finding

Use a statement another engineer can verify:

09:42 UTC from web-03:
- DNS returned 203.0.113.25 as expected.
- The kernel selected eth0 through 10.20.4.1.
- TCP 443 completed in 18 ms.
- TLS succeeded for api.example.com.
- The load balancer returned HTTP 502.
- The backend health check on 127.0.0.1:3000 was refused.

This points to the backend process or service configuration without changing DNS, routes, or public firewall rules.

References

• ss socket inspection
• ip-route lookup
• curl diagnostics