Troubleshooting & Ops Log

Symptom: Users started hitting 504 Gateway Timeout errors intermittently on a production web app fronted by Nginx. No deployments had gone out recently, and the errors weren't consistent — some requests succeeded, others timed out after ~60 seconds.

Root cause: Nginx was acting as a reverse proxy to an upstream application service. The upstream service had entered a degraded state — its worker processes were alive but no longer accepting new connections, likely due to a connection pool exhaustion issue on the app side. Nginx correctly waited for a response, hit its proxy_read_timeout, and returned a 504 to the client.

Fix:

• Confirmed Nginx's error log (/var/log/nginx/error.log) showed repeated upstream timed out entries, ruling out Nginx config as the cause.
• Checked the upstream service with systemctl status app-service — process was running but not responding.
• Restarted the hung upstream service: systemctl restart app-service, clearing the stuck connections.
• Reloaded Nginx (systemctl reload nginx) to drop any queued requests pointed at the old worker.
• Lowered proxy_read_timeout and enabled active health checks on the upstream block.

Result: 504 errors stopped immediately. Added a monitoring alert on upstream response latency to catch this before it reaches users next time.

Symptom: A Windows Server-hosted site started throwing HTTP 404 — Page Not Found for every request, including static content that hadn't changed in months. A straightforward iisreset did not resolve it — the site came back up briefly, then failed again within seconds.

Root cause: IIS's World Wide Web Publishing Service (W3SVC) depends on the Windows Process Activation Service (WAS) to manage application pools and worker processes. WAS itself had entered a stopped/faulted state, likely due to a corrupted application pool configuration change made earlier that day.

Fix:

• Checked Windows Services (services.msc) and confirmed both W3SVC and WAS were in a stopped/erratic state.
• Stopped both services manually and explicitly, one at a time — instead of a blanket iisreset.
• Started WAS first, since W3SVC depends on it being healthy.
• Once WAS reported "Running," started W3SVC next.
• Verified the affected application pool had recycled cleanly in IIS Manager before testing the site.

Result: Site came back up immediately and stayed stable. Lesson: when iisreset doesn't hold, check WAS specifically — the dependency order (WAS → W3SVC) matters.

Symptom: A subset of EC2 instances in a private subnet began failing outbound DNS lookups intermittently, causing application timeouts during peak traffic.

Root cause: VPC DNS resolver was hitting the default 1024 packets/sec per-ENI limit under high concurrent lookup volume, silently dropping excess queries.

Fix:

• Deployed a local DNS caching layer (dnsmasq or systemd-resolved) on affected instances to reduce upstream query volume.
• Split high-traffic services across multiple ENIs to distribute DNS query load.
• Added CloudWatch alarms on VPC DNS query metrics to catch this proactively going forward.

Result: Timeout errors dropped to zero after rollout; verified over a full week of peak traffic cycles.