Using serial console as a poor mans IP kvm?
I want to set up a few servers at home. Unfortunately, as I live in Bulgaria at the moment, the electric power is gone pretty often for longer periods than my UPS'es can deal with. So my servers will have to be started at least a few times every quarter. Another challenge with living in Bulgaria is that there is no law or order. The Police is just a branch of the Mafia. I need to protect the data on the servers with full disk encryption in case they are stolen. That means that I need to reboot the servers relatively often, and provide the luks passwords every time. Some times I am far away when this happens. I have been considering Supermicro motherboards with built in support for remote management - or old KVM IP switches from Ebay. The problem with Supermicro is that it's expensive and difficult to get the RAM required for their recent Skylake boards. The problem with Ebay is that few suppliers ships to Bulgaria, and getting anything trough the custom's here takes a whole day. Then there is the question if the device works at all... So I'm thinking about serial consoles. My gateway router will reboot after an outage, and it can act as a VPN endpoint. So I can access IP devices. With a rasberry pi and some relays, I can probably trigger a cold reboot whenever I need to. If I could log on to the grub console on the servers over a serial link, that's all I need, really. Does anyone here have any experience with remote control with Debian boxes over serial? Will it work reliable? Thanks in advance. Jarle
Re: Using serial console as a poor mans IP kvm?
On Thu, Sep 08, 2016 at 10:26:59PM +0300, Jarle Aase wrote: > I want to set up a few servers at home. Unfortunately, as I live in Bulgaria > at the moment, the electric power is gone pretty often for longer periods > than my UPS'es can deal with. So my servers will have to be started at least > a few times every quarter. > > Another challenge with living in Bulgaria is that there is no law or order. > The Police is just a branch of the Mafia. I need to protect the data on the > servers with full disk encryption in case they are stolen. > > That means that I need to reboot the servers relatively often, and provide > the luks passwords every time. Some times I am far away when this happens. I > have been considering Supermicro motherboards with built in support for > remote management - or old KVM IP switches from Ebay. The problem with > Supermicro is that it's expensive and difficult to get the RAM required for > their recent Skylake boards. The problem with Ebay is that few suppliers > ships to Bulgaria, and getting anything trough the custom's here takes a > whole day. Then there is the question if the device works at all... > > So I'm thinking about serial consoles. My gateway router will reboot after > an outage, and it can act as a VPN endpoint. So I can access IP devices. > With a rasberry pi and some relays, I can probably trigger a cold reboot > whenever I need to. If I could log on to the grub console on the servers > over a serial link, that's all I need, really. > > Does anyone here have any experience with remote control with Debian boxes > over serial? Will it work reliable? We use serial consoles on Debian and Oracle Linux boxes all the time, and have done so for more than a decade. They are more dependable than anything else -- once you have set them up and tested them through a full reboot cycle. There are relatively expensive, but compact, devices that will do both serial access and power switching. 8 of each is a common configuration, as is 16. We like WTI boxes. I am somewhat suspicious of USB-to-serial adapters in general, but they are cheap and you can hook up lots at once through a USB hub. You will probably want to test several brands in order to find something reliable. Incidentally, there are very few applications in which a Skylake processor will be notably faster than the previous generation of Broadwells -- and Broadwells can use DDR3. You might save a lot of money and get built-in KVMs that way. -dsr-
Re: Using serial console as a poor mans IP kvm?
On 9/8/16 3:26 PM, Jarle Aase wrote: I want to set up a few servers at home. Unfortunately, as I live in Bulgaria at the moment, the electric power is gone pretty often for longer periods than my UPS'es can deal with. So my servers will have to be started at least a few times every quarter. Another challenge with living in Bulgaria is that there is no law or order. The Police is just a branch of the Mafia. I need to protect the data on the servers with full disk encryption in case they are stolen. That means that I need to reboot the servers relatively often, and provide the luks passwords every time. Some times I am far away when this happens. I have been considering Supermicro motherboards with built in support for remote management - or old KVM IP switches from Ebay. The problem with Supermicro is that it's expensive and difficult to get the RAM required for their recent Skylake boards. The problem with Ebay is that few suppliers ships to Bulgaria, and getting anything trough the custom's here takes a whole day. Then there is the question if the device works at all... So I'm thinking about serial consoles. My gateway router will reboot after an outage, and it can act as a VPN endpoint. So I can access IP devices. With a rasberry pi and some relays, I can probably trigger a cold reboot whenever I need to. If I could log on to the grub console on the servers over a serial link, that's all I need, really. Does anyone here have any experience with remote control with Debian boxes over serial? Will it work reliable? It sort of works. I've done this two ways: 1. External serial-to-ethernet box. The external box turns out to be somewhat flakey, and a security hole (unpatched embedded linux with some vulnerabilities, and it needs to be rekeyed annually, but that doesn't actually work very smoothly). 2. Supermicro IPMI board: Sometimes works, sometimes simply doesn't respond - usually when one needs it most. In both cases, unless you layer a VPN on top of them, they are really nasty security holes. I've ended up resorting to the old "call the data center and have a human push the button" - but that doesn't sound like it applies to your situation. Good luck finding a solution. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra
Re: Using serial console as a poor mans IP kvm?
On Thu, 8 Sep 2016 22:26:59 +0300 Jarle Aase wrote: > I want to set up a few servers at home. Unfortunately, as I live in > Bulgaria at the moment, the electric power is gone pretty often for > longer periods than my UPS'es can deal with. So my servers will have to > be started at least a few times every quarter. > > Another challenge with living in Bulgaria is that there is no law or > order. The Police is just a branch of the Mafia. I need to protect the > data on the servers with full disk encryption in case they are stolen. > > That means that I need to reboot the servers relatively often, and > provide the luks passwords every time. Some times I am far away when > this happens. I have been considering Supermicro motherboards with built > in support for remote management - or old KVM IP switches from Ebay. The > problem with Supermicro is that it's expensive and difficult to get the > RAM required for their recent Skylake boards. The problem with Ebay is > that few suppliers ships to Bulgaria, and getting anything trough the > custom's here takes a whole day. Then there is the question if the > device works at all... > > So I'm thinking about serial consoles. My gateway router will reboot > after an outage, and it can act as a VPN endpoint. So I can access IP > devices. With a rasberry pi and some relays, I can probably trigger a > cold reboot whenever I need to. If I could log on to the grub console on > the servers over a serial link, that's all I need, really. > > Does anyone here have any experience with remote control with Debian > boxes over serial? Will it work reliable? Generally speaking I haven't used a serial console on Debian in particular. I do it on a Linux-based system I maintain; serial console works very well, provided you remember the main differences: it's not a VESA console, it doesn't know about CTRL-ALT-DEL, and you may see no output until grub starts (unless the BIOS/firmware can do serial console). The system's terminal type may not match the emulator's type and the display may be somewhat garbled; the use of serial ports for interactive use has declined greatly over the years, as has conformance to serial terminal protocols. Also, a starts SysRq. I suspect grub should work with a serial console; I've never tried it with the new grub. I wrote a 6-line patch for grub legacy (which I use for my system) that allows one to use either VESA or serial console, or choose one with a keystroke. I've been using it for 3-4 years now without trouble. To tell linux to use a serial console, connect a terminal (or emulator) to ttyS0 (COMa) on the server and set it to 115200-8-N-1, let grub start, then edit the boot entry and add the option "console=ttys0,115200" to the kernel command line, and boot. Your main problem will be operating the servers' reset switches should it be necessary. I think this is usually done by having the DCD, DSR, RI, or CTS line of the TIA-232 port close-then-open a relay that acts as the reset switch. (But your idea of using an rPI would work, too.) Of course, the encryption key reader has to work on a serial port as well. What would be interesting is if there were a 'VGA scanner' for the rPI so it could send you the screen changes, say, ten times a second. (And there are some USB devices that may work.) And if the rPI had a client USB port (so it could act like a keyboard) you would be able to see the 'monitor' and type on the 'keyboard'.
Re: Using serial console as a poor mans IP kvm?
For remote access, the RPi sounds like a good idea to me. I've had one on the 'Net for several years, doing things not requiring major CPU power. It's on my UPS, and it's had no reliability problems. A relatively small dedicated UPS would likely keep your border router and an RPi going for quite a while. An RPi is just a small Debian box, so you could reliably get to it over a VPN. Then, with a little innovative routing, you could look around the net(s) and see what's going on. With a relay board on the RPi's GPIO connector, you could turn things back on, if necessary. No RS-232, new motherboards, or KVM anythings required. Assuming you can write a little Python... No? -- Glenn English
Re: Using serial console as a poor mans IP kvm?
On Thu, 8 Sep 2016 15:43:31 -0600 Glenn English wrote: > For remote access, the RPi sounds like a good idea to me. I've had one on the > 'Net for several years, doing things not requiring major CPU power. It's on > my UPS, and it's had no reliability problems. > > A relatively small dedicated UPS would likely keep your border router and an > RPi going for quite a while. > > An RPi is just a small Debian box, so you could reliably get to it over a > VPN. Then, with a little innovative routing, you could look around the net(s) > and see what's going on. > > With a relay board on the RPi's GPIO connector, you could turn things back > on, if necessary. No RS-232, new motherboards, or KVM anythings required. Except for entering the encryption keys, and seeing when to enter them
Re: Using serial console as a poor mans IP kvm?
On Thu, 08 Sep 2016, Jarle Aase wrote: > Does anyone here have any experience with remote control with Debian boxes > over serial? Will it work reliable? It's fairly reliable; I actually prefer it to using KVM in almost all cases. You just need to get it configured properly in grub, the bios, and the kernel, and you should be fine. That said, providing LUKS input over the wire is always going to be problematic unless you have known secured links to the terminal. [But maybe you'll know if the government has done this to you.] -- Don Armstrong https://www.donarmstrong.com Whatever you do will be insignificant, but it is very important that you do it. -- Mohandas Karamchand Gandhi
Re: Using serial console as a poor mans IP kvm?
On 09/08/2016 10:26 PM, Jarle Aase wrote: >... > So I'm thinking about serial consoles. My gateway router will reboot > after an outage, and it can act as a VPN endpoint. So I can access IP > devices. With a rasberry pi and some relays, I can probably trigger a > cold reboot whenever I need to. If I could log on to the grub console on > the servers over a serial link, that's all I need, really. > > Does anyone here have any experience with remote control with Debian > boxes over serial? Will it work reliable? Quite a while back (Etch) I had some Debian machines running via serial console. As far as I know everything should still work just as nicely over serial console. From what I recall, you'll have to set console settings several places in the system to cover all contingencies for booting and recovery. I've used USB-to-serial adapters with the Prolific chipset. They've worked fine for me, in various models. (I haven't tried FTDI and am suspicious of them.) There are also specialized PCI and PCIe serial console servers which add 4 or 8 extra serial ports to a machine. But if you're going to run everything off of a single rpi then a USB-to-serial adapter is the way to go. There are ones that go USB to 4 or 8 serial ports, but they are hard to find affordably any more. About the power relays, I did that before and had a lot of help to make some custom ones, nothing being on the market back then. I found someone with skill to build a custom setup that worked over GPIO. However, nowadays there are several devices that look interesting. One pre-made series that caught my eye a few weeks ago was this one: https://unipi.technology/shop/ However, I have not evaluated any units so that is just to point to what's on the market and not any endorsement. You'll need to wire plugs and such, too, and I can't see any fuses on those units. Regards, Lars
Re: Using serial console as a poor mans IP kvm?
On Fri, 2016-09-09 at 08:46 +0300, Lars Noodén wrote: > I've used USB-to-serial adapters with the Prolific chipset. They've > worked fine for me, in various models. (I haven't tried FTDI and am > suspicious of them.) And my experience is the opposite. I have genuine (there's apparently a lot of fakes) FTDI devices in pretty much daily use for many years without problems. This is using ser2net on a local network for accessing serial consoles on ARM based development boards. ser2net will be insecure telnet or raw port forwarding but if it's not exposed to the internet and you can ssh tunnel into the local network then that's a lot better. I've done that method for carrying on working with my boards whilst across the other side of the world. Of course, a means of power cycling devices is also essential. -- Tixy
Re: Using serial console as a poor mans IP kvm?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, Sep 08, 2016 at 10:26:59PM +0300, Jarle Aase wrote: > I want to set up a few servers at home. Unfortunately, as I live in > Bulgaria at the moment, the electric power is gone pretty often for > longer periods than my UPS'es can deal with. So my servers will have > to be started at least a few times every quarter. [...] > That means that I need to reboot the servers relatively often, and > provide the luks passwords every time. Some times I am far away when > this happens [...] An interesting alternative to the serial console thing is baking in an SSH server into the initramfs. There are small SSH servers built for that, like Dropbear. Upside is that you don't need any additional hardware and it's pretty well integrated into Debian. Downside is that you need BIOS, the bootloader and initramfs working (with the serial you at least get a chance to fix the bootloader remotely). https://packages.debian.org/sid/dropbear-initramfs https://wiki.debian.org/RescueInitramfs https://projectgus.com/2013/05/encrypted-rootfs-over-ssh-with-debian-wheezy/ Might be worth a try. Regards - -- t -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlfSZVMACgkQBcgs9XrR2kZGNgCfZhrMlouUceQSVJgzimE+b2YG GokAn0PpEqnw2lgmFiGTu554OQtpt9Wa =AKQd -END PGP SIGNATURE-
Re: Using serial console as a poor mans IP kvm?
On Freitag, 9. September 2016 08:15:37 PYT Tixy wrote: > On Fri, 2016-09-09 at 08:46 +0300, Lars Noodén wrote: > > I've used USB-to-serial adapters with the Prolific chipset. They've > > worked fine for me, in various models. (I haven't tried FTDI and am > > suspicious of them.) > > And my experience is the opposite. I have genuine (there's apparently a > lot of fakes) FTDI devices in pretty much daily use for many years > without problems. This is using ser2net on a local network for accessing > serial consoles on ARM based development boards. ser2net will be > insecure telnet or raw port forwarding but if it's not exposed to the > internet and you can ssh tunnel into the local network then that's a lot > better. I've done that method for carrying on working with my boards > whilst across the other side of the world. Of course, a means of power > cycling devices is also essential. I second that. I had many weird problems with PL2303 but never any on any OS* with FTDI FT232 chips. [*] Debian-Linux, OpenBSD, OSX and MS-Windows -- Eike Lantzsch ZP6CGE
Re: Using serial console as a poor mans IP kvm?
Hi, I was just about to order some usb2serial hardware when I read this. Your suggestion will give fewer "moving parts" and is actually very simple to implement. I will loose the ability to do a cold boot, but it will probably not matter too much in my case, at least not for now. I'll try it when I get the first server assembled. Thanks a lot! Jarle Den 09. sep. 2016 10:31, skrev to...@tuxteam.de: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, Sep 08, 2016 at 10:26:59PM +0300, Jarle Aase wrote: I want to set up a few servers at home. Unfortunately, as I live in Bulgaria at the moment, the electric power is gone pretty often for longer periods than my UPS'es can deal with. So my servers will have to be started at least a few times every quarter. [...] That means that I need to reboot the servers relatively often, and provide the luks passwords every time. Some times I am far away when this happens [...] An interesting alternative to the serial console thing is baking in an SSH server into the initramfs. There are small SSH servers built for that, like Dropbear. Upside is that you don't need any additional hardware and it's pretty well integrated into Debian. Downside is that you need BIOS, the bootloader and initramfs working (with the serial you at least get a chance to fix the bootloader remotely). https://packages.debian.org/sid/dropbear-initramfs https://wiki.debian.org/RescueInitramfs https://projectgus.com/2013/05/encrypted-rootfs-over-ssh-with-debian-wheezy/ Might be worth a try. Regards - -- t -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlfSZVMACgkQBcgs9XrR2kZGNgCfZhrMlouUceQSVJgzimE+b2YG GokAn0PpEqnw2lgmFiGTu554OQtpt9Wa =AKQd -END PGP SIGNATURE-
Re: Using serial console as a poor mans IP kvm?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, Sep 09, 2016 at 09:03:33PM +0300, Jarle Aase wrote: > Hi, > > I was just about to order some usb2serial hardware when I read this. [...] > I'll try it when I get the first server assembled. Thanks a lot! Hey, glad to help :-) - -- t -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlfTBAQACgkQBcgs9XrR2kb2oQCfXciqNGw+duZSi0+j293y9X26 mFcAnRJFuhTP641mbXmJ9YS0l5VZnd6j =3unC -END PGP SIGNATURE-
