Article · 3 September 2026 · 6 min read

Debug Detective: the phantom rule

The scene was set: Apache should have been listening on port 3002, but from my jump server the app was as good as dead. Every curl came back empty. The host was alive, but the service was nowhere to be found.

The scene

I sshed into the suspect machine to investigate. First step, check the service, and it came back failed. Apache wasn't running at all.

The logs gave me the first clue: address already in use: AH00072. Something else had already taken control of port 3002. So I asked who.

The suspect

sendmail. Not a service I was thinking about, holding a port I needed, bound to loopback only, which is exactly why Apache couldn't take 0.0.0.0:3002 and gave up.

Along the way I learned something worth more than the answer. Run that same command without my sidekick Sudo and the PID column goes blank:

I ended the impostor's run with a single bullet to the heart:

kill 452
#!Bang!

A little bit of CPR brought back my friend Apache, who was now able to take his rightful place at port 3002.

A curl from localhost reached port 3002 without issue. Was the case this simple?

No route to host

I went back to the office and curled from the jump server to check the job was done, but... No route to host. Unusual. Let's ping it.

That was strange. The server responded to pings. The path was open. Which meant the blockage came from something else. It had to be at the firewall level.

The false lead

I went back on the scene to take a look at iptables, and everything seemed fine.

The third rule allowed all traffic through, and it sat higher than the reject at the bottom, so the curl should have worked. The only odd thing on the whole screen was one little warning at the very end, about legacy tables being present.

So I chased it. I searched and searched for hours, trying to find these legacy tables, and they were nowhere near. After a long reflection I decided it would be best to take a step back and reanalyse the situation objectively, without going down this route.

That is the part of this case worth keeping. The clue was real, it was printed on my screen by the tool itself, and it had nothing to do with my problem. A clue you can see is not the same as a clue that matters.

The reveal

That's when it hit me. I remembered that a long time ago a colleague solved a similar case using one little helper. Add -v.

There it is. Plain -L does not print the interface columns. -L -v does. Rule 3 wasn't letting everything through at all. It was letting everything through on the loopback interface.

My packets were knocking on the wrong door, and iptables just shrugged.

That is also why a curl from localhost worked and a curl from the office didn't. The packet counts say the same thing out loud: rule 3 had matched a single packet, and the reject at the bottom had matched three. Those three were mine.

One rule, and the door opened:

sudo iptables -I INPUT -p tcp -s didi_office -d 172.16.238.10 --dport 3002 -j ACCEPT

Detective's note

Two suspects, one case. A process I wasn't thinking about, and a rule that said one thing in the summary and another in the detail. Neither was hiding. I just wasn't asking either of them the right question.

• Always use -v with iptables. The packet counts and the interface columns are where the truth lives.
• Never assume a rule applies globally. Check the in column before you trust it.
• Without sudo, netstat hides the PID column. The name of your suspect is one word away.
• When a clue costs you hours and leads nowhere, suspect the clue, not the case.
• "It works from localhost" is a result about localhost. It is not a result about anybody else.