Troubleshooting with the Usual Suspects

When something breaks in production, the worst thing I can do is guess randomly. Over the years I’ve settled on a short list of “usual suspects” I interrogate in order. It’s not glamorous, but it finds the culprit far more often than intuition does.

1. What changed?

Most incidents follow a change. Before touching anything else, I ask: was there a deployment, a config update, a certificate rotation, a firewall rule, a patch window? If the answer is yes, I have my prime suspect. Rollback first, root-cause later — the business wants the service back, not a lecture.

2. What do the logs say?

Logs rarely tell the whole story, but they usually point in a direction. I start broad and narrow down:

# Recent errors across the system
journalctl --since "30 min ago" -p err

# Follow the application log while reproducing the issue
tail -f /var/log/myapp/app.log | grep -iE "error|timeout|refused"

# Did the service restart or get killed?
journalctl -u myapp --since today | grep -iE "start|stop|kill|oom"

An OOMKilled or Connection refused at this stage saves an hour of speculation.

3. Resource exhaustion

Slow is often just “full”. Quick checks: df -h for disk (don’t forget inodes with df -i), free -h for memory, top or htop for CPU and load, and ss -s for socket counts. A full disk or an exhausted connection pool explains a shocking percentage of “the app is weird” tickets.

4. Dependencies

My service might be fine — the database, message queue, API gateway, or third-party endpoint behind it might not be. I test each hop explicitly with curl -v, nc -zv host port, or the client’s own health check. Timeouts point to network or load; instant refusals point to a down service or a firewall.

5. It’s always DNS

Half joke, half law of nature. Stale records, expired zones, a broken resolver in /etc/resolv.conf, or a container using a different DNS path than the host. dig +short service.internal from the affected machine, compared against a known-good resolver, settles it quickly.

The takeaway

A checklist beats cleverness under pressure. Changes, logs, resources, dependencies, DNS — walk the list calmly, note what you ruled out, and the incident usually confesses on its own.