From the blog.

Managing Digital Racket
The more I tune out, the less I miss it. But that has presented me with some complex choices for a nuanced approach to curb
Complexity – My Friend, My Enemy
Over my years of network engineering, I've learned that the fewer features you can implement while still achieving a business goal, the better. Why? Fewer

Opengear IM4200 ‘Connection Refused’ on SSH to Ports 3001+

896 Words. Plan about 5 minute(s) to read this.

Opengear makes out-of-band management console servers with a great deal of flexibility. In essence, an Opengear box is a Linux machine with ports that can act in a variety of roles. Various models have various console port densities and specific functions.

I ran into an issue with an Opengear IM4216-34 running firmware 3.5.3u1 where the box would give me an immediate “connection refused” when attempting to SSH to ports 3001 and up after it had been working for weeks, and with no configuration changes. You can configure an Opengear so that opening an SSH connection to a specific port number maps you directly through to a specific console port. This is a shortcut to logging into the Opengear CLI with “username:serial”, and then connecting to the specific port you need. TCP port 3001 maps to console port 1, 3002 to console port 2, etc.

The symptom I would see was a straight up “connection refused” – as if I was getting a TCP reset back from the Opengear. After a reboot of the unit didn’t resolve it, I opened a case with Opengear support, and we went through the following steps to prove out the issue.

1. Check iptables. We had no iptables policy installed on this particular Opengear.

iptables -Z  # resets the iptables counters, make sure you’ll get the log output you’re looking for.

tail -f /var/log/messages  # displays text added to the /var/log/messages file until you break out with CTRL-C. If iptables is blocking the connection, you’d see an appropriate message.

2. Run tcpdump to grab a few packets. I actually did this on my own, because I wanted to verify that the Opengear was generating a TCP reset, and not some intermediate device. 192.168.250.191 is my client, and 192.168.251.10 is the Opengear unit.

The first packet shows my SSH client generating an inbound SYN.

The second packet shows a RST coming back immediately.

# tcpdump -i eth0 -nn ‘host 192.168.250.191 and not port 22’
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:21:42.923025 IP 192.168.250.191.62059 > 192.168.251.10.3001: S 1215835747:1215835747(0) win 65535 <mss 1240,nop,wscale 4,nop,nop,timestamp 16859700 0,sackOK,eol>
13:21:42.923025 IP 192.168.251.10.3001 > 192.168.250.191.62059: R 0:0(0) ack 1215835748 win 0
^C
2 packets captured
4 packets received by filter
0 packets dropped by kernel
#

3. Verify that the listeners are active. We’d ruled out the most likely culprit – iptables. We’d proven the Opengear was indeed sending a TCP reset to terminate the connection. So, now it was down to seeing if the port listeners for 3001+ were even spun up. In the output below, we should have seen netstat report listeners on various port 300x sockets, but we didn’t.

# netstat -an | grep 300
tcp 0 0 127.0.0.1:32000 127.0.0.1:53009 ESTABLISHED
tcp 0 0 127.0.0.1:53009 127.0.0.1:32000 ESTABLISHED

4. Verify that the configuration would even spin up the listeners. We knew the listeners weren’t there. But why not? Here, in inetd.conf, we don’t see several “ssh_serial” directives that should be there.

# cat /etc/config/inetd.conf
telnet stream tcp6 nowait root /bin/telnetd
#

5. Run magical Opengear config scripts to set things right. With the problem identified, the support engineer rode in on a unicorn to resolve. I don’t remember what “bingo” followed by the “yes it did” output was all about, but it was meaningful to Opengear.

# config -g config.ports | grep ssh
config.ports.port1.ssh on
config.ports.port2.ssh on
config.ports.port3.ssh on
config.ports.port4.ssh on
config.ports.port5.ssh on
config.ports.port6.ssh on
config.ports.port7.ssh on
# config -r serialconfig
– Configurator ‘ups’ ran in 0 seconds.
– Configurator ‘enviro’ ran in 0 seconds.
– Configurator ‘services’ ran in 1 seconds.
– Configurator ‘serialconfig’ ran in 0 seconds.
– Configurator ‘console’ ran in 0 seconds.
– Configurator ‘dialin’ ran in 0 seconds.
– Configurator ‘users’ ran in 0 seconds.
– Configurator ‘secrets’ ran in 1 seconds.
– Configurator ‘modemwatchdog’ ran in 0 seconds.
– Configurator ‘hosts’ ran in 0 seconds.
– Configurator ‘nagios’ ran in 0 seconds.
– Configurator ‘power’ ran in 1 seconds.
– Configurator ‘cascade’ ran in 0 seconds.
– Configurator ‘sshconfig’ ran in 0 seconds.
– Configurator ‘powersupplies’ ran in 0 seconds.
– Configurator ‘snmp’ ran in 0 seconds.
– Configurator ‘sshforwards’ ran in 0 seconds.
– Configurator ‘stunnel’ ran in 0 seconds.
# cat /etc/config/inetd.conf
telnet stream tcp6 nowait root /bin/telnetd
ssh_serial01 stream tcp6 nowait root /bin/pminetd -l port01 –ssh
ssh_serial02 stream tcp6 nowait root /bin/pminetd -l port02 –ssh
ssh_serial03 stream tcp6 nowait root /bin/pminetd -l port03 –ssh
ssh_serial04 stream tcp6 nowait root /bin/pminetd -l port04 –ssh
ssh_serial05 stream tcp6 nowait root /bin/pminetd -l port05 –ssh
ssh_serial06 stream tcp6 nowait root /bin/pminetd -l port06 –ssh
ssh_serial07 stream tcp6 nowait root /bin/pminetd -l port07 –ssh
# netstat -an | grep 300
tcp        0      0 :::3001                 :::*                    LISTEN
tcp        0      0 :::3002                 :::*                    LISTEN
tcp        0      0 :::3003                 :::*                    LISTEN
tcp        0      0 :::3004                 :::*                    LISTEN
tcp        0      0 :::3005                 :::*                    LISTEN
tcp        0      0 :::3006                 :::*                    LISTEN
tcp        0      0 :::3007                 :::*                    LISTEN
#
# bingo
# yes it did

Conclusion

There was no good explanation from Opengear as to what caused this problem. This appears to be “one of those things.” Also, I’m not 100% sure it’s the right thing to run the magic scripts on your own without Opengear saying so. I have no sense of their overall impact to the configuration. Opengear devices are extremely capable, with a lot of connectivity options. My guess is that it’s possible to screw up some part of your configuration playing with scripts like this if you don’t know what you’re doing. YMMV.

With any luck, this post will help someone else.