Geek pop quiz!

[theweaselking]

I fell incredibly stupid today.

Ubuntu server. Running a brand new Perforce server, listening on port 1666.

Perforce clients on the server can see the Perforce server.
Perforce clients anywhere else cannot connect. They *could* connect very briefly when I first started the server but now they can't.[1] No, I didn't change anything.[2]
Nmap from the server says port 1666 is open on the server.
Nmap from anywhere else says port 1666 is not open on the server.

Telnet confirms this: It's listening to 1666 from localhost but not from anywhere else.

There is no software firewall running on the server. The hardware firewall is set to forward port 1666 from any LAN address to the server.

This has to be an incredibly simple thing. What the FUCK am I doing wrong, here?


[1]: I could connect as "Perforce", the superuser, because that was what I tried first. I went to connect as a different user, but before I did that the thing *stopped listening on that port at all*.

[2]: No, really. Literally I confirmed that the superuser worked, clicked "back" to change the username to a different user, and connections no longer worked at all.


EDIT: Changed the fucking port and it works. What the fucking fuck is wrong with fucking 1666 that isn't fucking wrong with 11666?

Comments

[elffin.livejournal.com]

I'd change the physical port connection first.

Then I'd move it to a different backbone.

Then I'd change out the NIC.

I'm going to /guess/ that there's a broken rule, or a hidden rule, somewhere on the hardware firewall that's killing 1666 - like, a traffic squelch rule or a QoS rule or an IDS rule that the Perforce traffic is tripping. or something.

[theweaselking.livejournal.com]

One single TCP port fails while all others are working on a known-good currently running server, and you're thinking *hardware*?

[elffin.livejournal.com]

I'm actually thinking that there's a 90% chance that there's a firewall rule on the hardware firewall that isn't immediately obvious that's swallowing traffic.

The 10% chance is that the NIC is broken and doesn't like particular bit patterns and barfs on them, and TCP/UDP packets with a destination port of 1666 are the lucky winner of the "first to show up obviously as bad" lottery. This only because you mention that it worked /briefly/ and then not-at-all.

[ronebofh.livejournal.com]

Are they on the same network? Is the server's netmask set correctly?

Also, check the default route. See if the perforce server settings are telling it to only bind to localhost. Also, is it listening only on localhost or is it also listening on the host address?

[theweaselking.livejournal.com]

All other ports for all other services between the clients and the server are working. The mask is set correctly.

Perforce server settings are *not* set to bind only to localhost.

Local clients *can* connect by IP and servername as well as "localhost".

And changing the port worked, but I don't know why.

[ronebofh.livejournal.com]

Hoo, good one. Switch it back and see if it works now, i guess.

[kadath.livejournal.com]

I would power cycle it a couple of times, then give up and whine for IT to come fix it!

[theweaselking.livejournal.com]

Yeah, unfortunately that's *me*.

[kadath.livejournal.com]

*hovers*

I really need to get this file uploaded!

...I already tried that!

What's taking so long!

[theweaselking.livejournal.com]

As soon as you start paying my hourly rate, I will happily help you with all your needs.

[demiurgent.livejournal.com]

Eight hundred and six jokes, most of them inappropriate, immediately spring to mind for the followers of the thread....

[xomox.livejournal.com]

I'd suspect that selinux was enabled, since I've seen similiar mysteries on fresh redhat/centos boxes. But then, you say there's no software firewall enabled.

[jsbowden.livejournal.com]

SELinux isn't a firewall, but all the same, I either turn it off completely (normally) or set it to non-enforcing since it's a cryptic POS with no docs, no decent interface, and no way to debug that doesn't involve kafka-esque bullshit. And yeah, check to see if it got enabled. I've seen it randomly show up after an update before. I hate that shit. Please don't fuck with my settings just because you installed a x.xxxxN update to some binary somewhere that this touches.

[zastrazzi.livejournal.com]

What does lsof -i TCP show on the perforce server?

Any hints in /var/log/messages or syslog? Anything useful in the perforce logs?

If memory serves from about 2 years ago, the binary license also defines a number of server parameters on start.

[demiurgent.livejournal.com]

My basic troubleshooting would be to start by swapping out the ethernet cord and plug it into a different port on the switch or (by preference) a different switch entirely. A broken ethernet cord can manifest by causing interference with certain data patterns which can manifest as 'blocking a single port.' It's not as likely if just changing the port fixed it, but it's possible and it's simple to test.

The port and/or switch, on the other hand, is entirely possible depending on the complexity of said switch. A munged ARP table or cache, misconfiguration, a failing physical port -- all of these things can cause certain protocols and ports to get blocked. If the switch is complicated enough, then it could be kicking port 1666 into a nonexistent VLAN.

If, on the other hand, your switches are pretty stupid -- or if God help you your network uses hubs -- then check to see if you have a broadcast storm on your LAN. Give enough traffic, and you can see specific ports end up being rejected even as others pass straight through.

If on the other other hand you're essentially not doing LAN traffic at all, but instead are just passing through a router to the internet, check to see if your ISP interdicts port 1666 for some reason. I know Rogers does weird crap sometimes.

[elffin.livejournal.com]

So, is there any actual explanation of the weirdness forthcoming, or is it still an unknown factor and will remain an unknown as to what caused the problem in the first place?

[theweaselking.livejournal.com]

It's one hop away, and I can reach it just fine. IT's just not listening on that one port.

Settings: All correct, which is the problem.
Manual: Hahahahahahahaitistolaugh.
Firmware: For *what*? Neither client nor server is an embedded device. This is software!

(Something else on port 1666: No, I checked that. But that's a good thought!)

[theweaselking.livejournal.com]

Protocol mismatch would mean that telnet on the port would work, I'd just get garbage.

Just a guess as to why 11666 is better than 1666.

[quotation.livejournal.com]

Perforce is an "Enterprise" application.

We know that ports below 1024 are reserved for superuser binding on some OSen.

1024 is a 4-digit number.

So, in our Enterprise application, we should block use of any port numbers that are 4 or fewer digits, just to be safe.

And then not bother to document it, because only a certified professional would be using Enterprise software.

Re: Just a guess as to why 11666 is better than 1666.

[theweaselking.livejournal.com]

Except that 1666 is the default port and changing it is actually, if not a pain in the ass, at least a little annoying.

[anthonybaxter.livejournal.com]

Sounds like it's listening on localhost:1666, rather than *:1666. What does netstat -na | grep 1666 show?

tcp ports

[maridonner.livejournal.com]

Hi, for tcp ports will better this - find port by description (http://ports.my-addr.com/tcp_port_list-udp_port_list_search_by_description.php) tool.