So Time Warner updated the infrastructure upstream of Mary’s folks and installed a new cable modem / router, which killed my remote access using ssh (with RSA keys, passphrases, nonstandard ports, fixed internal IP addresses, port forwarding, port triggers, and all the Right Stuff). I just spent a harried pair of days trying and failing to figure out how to make this work again.
My laptop can ssh into my file server from our house network, both wired and wireless. Ditto when it’s on the Squidwrench Operating Table. Ditto from the low-quality Hampton Inn WiFi near her folks. Plunked on their desk and jacked into their router, however, that outbound ssh times out somewhere between their bits and my basement.
I dinked with the TW Surfboard modem / router, added the appropriate port forwarding & triggers, dialed back the firewall intensity, and ssh flat out doesn’t work in either direction from any PC (all running various Linus flavors). No diagnostics, no logs, nothing that I could find.
From the outside (our house or the Hampton), there’s no response from the PCs inside (on their desk). I’m not trying a loopback from inside to inside, which I know doesn’t work with consumer-grade routers. I’d planned to ssh from there to my basement file server, then ssh back to verify that the connections worked, but the outbound connection doesn’t work.
Probably unrelated, but equally frustrating: trying to configure Thunderbird’s outbound SMTP with their email server flat-out doesn’t work. Either the username / password isn’t valid (it is), various combination of ports / security / encryption (including the ones in the TW FAQs) don’t survive the configuration test, or a seemingly valid configuration doesn’t actually transmit email. Incoming email works only in IMAP mode, not POP3.
I finally set up outbound TW email to bankshot through his Gmail account, which will probably have unforeseen side effects.
The usual Google searches were unavailing, other than several notes suggesting that if you have any other choice of ISP or email provider than TW, do that. But it’s not like they have any choice; Verizon provides 1 Mb/s (!) DSL in that area and satellite Internet isn’t going to happen in an apartment.
Obviously, I’m doing several things wrong, but I have no idea what else to try. I’ve set up email and remote access often enough to get a whole bunch of things right, but that sure didn’t help with TW.
Any suggestions?
The Smell of Molten Projects in the Morning 
If I’m hearing you right, you can’t SSH out of the in-laws house to things you can SSH to from anywhere else. If I got that right, let’s figure out whether the carrier is blocking the SSH port, the SSH protocol, or both.
[day job]Test the port: From in their house, do
telnet your-fileserver 22to try to open a TCP connection to your server’s SSH port. If it connects and you get something like
Connected to your-fileserverEscape character is '^]'.
SSH-2.0-OpenSSH_4.2
then the SSH port is open and they’re messing with you at an application level. If you get an immediate rejection, something is sending back a TCP reset, which is uncommon for ACLs and firewalls and usually indicates that the destination server doesn’t want to talk to you (so I don’t expect this to happen). If you get a long timeout, a firewall (usually in between, but can also be the destination server’s host firewall) is silently discarding your traffic without the courtesy of sending you a rejection notice. Of course these are heuristics, but they’re accurate the vast majority of the time.
Test the protocol: Can you reconfigure the firewall/router that protects your fileserver to forward inbound port 443 to the fileserver’s port 22 — in other words, plumb the HTTPS port to SSH? If so, then go somewhere else and test the new configuration by doing:
ssh -p 443 your-fileserverand assuming this works, go try the same from the in-laws.
I pick 443 because no matter how draconian the carrier, they’re going to allow HTTPS through. If this works from elsewhere and doesn’t work from the in-laws, then they’re doing application-layer filtering (“deep packet inspection”) to suss out that you’re running SSH and block it. If this does work from the in-laws, it confirms a hunch that the carrier is blocking port 22.
If you already have 443 plumbed because you’re running an HTTPS web server on the fileserver as well, you can try picking any arbitrary port for this test; but 443 is one that we can be particularly confident is permitted, so it’s a surer test.
If you get a chance to try some of those things, you can drop notes here or email me directly and we can talk about the next steps in diagnosis and/or workaround.
[/day job]You’ve got it right: plain old ssh from behind their router to a known-good destination on an unusual port doesn’t work, nor does ssh from here to there on that unusual port. I know better than to run a visible SSH server on port 22, even with public-key crypto, so they’re not blocking only that port.
Ping, of course, works fine, but that happens in the modem.
Firing telnet to the server in my basement on that unusual port works as you describe, but telnet to their box times out. I left their router set to direct that port to their static-addressed PC, with the ssh server listening to it, so telnet should connect.
Searching on the now-obvious keywords suggests that TW does, in fact, perform deep packet inspection as part of their traffic shaping. There’s obviously no hard proof.
We’re the better part of a three hour drive away (so I can’t just drop in) and voice-operated typing doesn’t work well (English wasn’t made to carry on that kind of conversation), so it’ll be quite a while before I get back to try anything else.
Thanks for the help!
If they’re doing DPI and messing with your SSH, it’s worth trying to set up a VPN — which may even be built into the modem/firewall. A friend couldn’t stream video directly over his DSL without it timing out; he VPNs from home to the office and streams HD over the VPN over the DSL without a glitch.
VPNs are important enough that most carriers aren’t blocking them yet. And yes, it’s possible to ascertain what’s running over the VPN by traffic analysis (packet sizes and timing) even though the content can’t be intercepted, but that’s not widespread yet.
Phew! Seems as though I must learn several new things in quick succession to get this done…
It’d be really great if we had actual ISP competition: I’d sign up for Google fiber in a heartbeat.
Re: the three-hour drive — if you have any kind of cell service that can run a wireless hotspot, on the next visit you can sit in the living room with one laptop on your hotspot, SSHed out to your destination system, and another laptop on their wireless, giving you access to both ends at the same time even when one isn’t cooperating.
Obviously you want to have done your homework first.
I have to play games like this … more often than I want to talk about.
Alas, we have an ancient & dirt-cheap Virgin Mobile flip phone that we never use and I’m pretty sure VM phones don’t support WiFi hotspots, either… [mutter]
Okay, that ate things I put into angle braces. Correcting only those lines:
Top of post: lt day job gt
telnet your-fileserver 22
Connected to your-fileserver
ssh -p 443 your-fileserver
The WP Comment Editor does a wonderful job of smashing anything that remotely resembles the (largely undocumented) WP markup tags, which reduces much of tech notation to gibberish. I put all the code-like things inside their code-in-angle-brackets tag, but there’s no obvious way to escape/quote/protect bare angle brackets.
Pfft!
My first thought was also port blocking. I’ve worked at companies that blocked all kinds of outgoing ports, including SSH (22). I did exactly what Keith suggested and ran SSH over port 443, which is pretty much never blocked. I was always going into my own server, so I either configured the server to use 443 or I configured my router to watch for that port and forward it to 22. Fortunately, the Internet is flexible enough that I can almost always find a way around unnecessarily intrusive IT departments.
I’ve also had ISPs block various SMTP and IMAP ports in the name of security. I’ve had to either switch to different ports that they do support or submit a form requesting an exception to have a port opened.
Remote support of family is a challenge no matter how you stack it.
Perhaps if I replace my blue-and-white engineer’s cap with an all-white painter’s cap, TW will let my packets through… [sigh]
For what it’s worth, I tried a search on ssh in the time-warner forums. One (partially) titled: Help-Port-Forwarding-Doesn-t-Seem-To-Work-Correctly might be useful. (He got it to work, eventually.)
Try forums [dot] timewarnercable [dot] com/t5/Internet/ct-p/twc_internet and search on ssh
I get the impression that TWC likes to do non-standard configurations of modems, and at least one manufacturer won’t/can’t support the model they supply to Time-Warner. Makes me happy with my satellite system, weather-related and solar-storm problems and all.
Sounds like I must force the modem into bridge mode and install my own downstream router, which will work until TW resets the modem. I don’t recall seeing “bridge mode” among all the setup options, but I wasn’t looking for that.
FWIW, I had a router downstream of the old modem, but removed it when the new modem-router arrived: no need for two routers in series, right? [sigh]
Good chance if you speak to the level 3 folks @ TW they will not only know what your talking about without having to explain much but also potentially help you get it corrected or debugged. Some of those folks are very helpful. I call them directly whenever there is a problem since I will have already gone through the initial troubleshooting (yes, my computer and modem are plugged in and on …).
Agreed, but punching through their automated voice interface and Levels 1+2 isn’t a task I’m willing to tackle during a holiday visit.
Perhaps I must rent a nearby apartment for a month, sign up for TW, spend a solid week getting my setup working, transfer the knowledge to their modem / router, and move out… [sigh]
I was lucky enough to have been given a local number to call. As you can imagine, I’ll never lose this! I think that if you start the process and simply ask for level three you can get it.
At some point we’ll be there during ordinary business hours and I’ll definitely give it a try!
Thanks for the tip!
I’d suggest more search-surfing. A trial search on SurfBoard modems (various models, sigh) points to several posts where somebody got it to work, with a list of the incantations and gyrations required to get it done. At least one procedure would have fit in a Harry Potter book, if Hogwarts had cable internet…
Your idea might work if you actually got the same hardware. I’d consider spending a week at their place (strategic vacation planning, perhaps). For what it’s worth, TWC likes to rent out the equipment, but there’s lists of compatible (maybe) equipment that you can buy. Not sure if that would increase or decrease the heartburn.
I’m a hell of a nice guy, but a man’s gotta know his limitations… [grin]
Maybe if you sent them on a vacation/day’s outing with Mary and did the work… I have some relatives best enjoyed in very small doses. [grin]
I had some similar problems with a new Comcast supplied gateway/wifi/router that Comcast required me to start using (had to do with my old modem being a single downstream channel, while their new modem load balances 8 channels) – installed thier device, killed my networks. Went the route of advanced tech support, and they applied some sort of override configuration to my router. The one problem is that the “fix” wouldn’t survive a reboot. My solution was simple – talk to their L1 tech support folks, ask them which 8 channel modem gives them the least trouble (note: modem, not integrated gateway/router/wifi). Bought that unit on Amazon, had Comcast activate it, and voila, no more trouble. The goal of my network is probably similar to what you seem to be doing – in my case, I have our primary residence networked together with our vacation house via an IPSec network. I used Ubiquiti Edgerouters ($99 on Amazon). Call me a control freak, but I like as few moving parts as possible “out of my control” where other, possibly conflicting, requirements/needs exist. I use plan old Netgear Wifi routers in bridge mode inside the networks, i.e. rather than using an integrated Wifi/router as the moat around the castle. The Ubiquiti has a bit of a learning curve to it. I’m familiar with Cisco IOS command line config stuff from some prior gigs, and the Ubiquiti is the same sort of text based config (for advanced applications – they have a GUI also).
Anyways, ditch the TWC provided box, and get a dumb modem. You’re dealing with less unknowns, and “tech support” is now under
your control.
If you do provide the modem, you may be able to get out of the rental fee that TWC is probably charging you.
Mmmmm, it gets complexier and complexier; I like your approach, though.
In principle, any DOCSIS 3.1 modem should work, but hell hath no fury like that of an unjustified assumption. If I were confident the Nice Voice on the phone would give me a straight answer, I’d be more confident about blowing away the modem; it’s obviously not something I should plan on doing over a long weekend.
Thanks for the suggestion!