Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
-Original Message- From: PLUG On Behalf Of Paul Heinlein Sent: Monday, February 27, 2023 12:06 PM To: Portland Linux/Unix Group Subject: Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown ) >In this hypothetical exchange, a business owner who didn't realize that a >single PC would make or break a significant customer relationship would in all >likelihood not be in business for very long. I have to disagree with this statement. There are TONS of smaller businesses where they do indeed pass up significant customer relationships all the time and yet still remain in business. It is just that doing this practically guarantees you will remain small. As a business owner if you CHOOSE to do this I don't have a problem with it. But you need to truly understand the risks. A 1 person business can easily pivot, and pivoting is needed when dancing with elephants. A 10 person company - not so much. They need to grab every opportunity. I've had a customer that size fail on me. They expanded too fast and collapsed when an elephant stomped on them. (and it was the largest elephant out there - the US Government) As a 10 person company they absolutely could not pivot. They are now a 1 person company - and pivoting - under a different name. Possibly diversifying might not have saved them given the industry they were in. But it might have given them a rathole to flee into. I dunno. >It's not that I disagree with your assessment, but I don't back off my initial >opinion that most IT decisions are based on risk assessment, not technology >>assessement. The latter can inform but will rarely trump the former. Buying an HP Elite or Pro notebook instead of their consumer grade stuff is not a technology assessment. It's a risk assessment. >Consider the business owner who is cash poor but relatively time rich. Keeping >cash on hand can justify the owner's need to spend extra time keeping a >>fragile set of systems working. I say "can," not "will" or "must," but I >think the point is reasonable. I used to think this way and indeed it's what motivated me to get into FreeBSD and Minix prior to that and TekUX prior to that. The idea was simple - as a poor young man I had lots of time and little cash, so why not spend the time learning this "free unix" stuff But what I learned is there's a point at which that tradeoff fails. I recall many years ago wanting to run the "screens" program on TekUX just because I wanted multiple screens on a dialup phone line to a box. I must have spent a month carefully teasing out how to compile it under TekUX until I finally ended up with a running binary. We don't use that program now because you can open multiple SSH or Telnet sessions to do the same thing. And even during the Dark Ages, I could have simply driven into the computer lab where the TekUX was located and done exactly the same thing. Of course, all of that was for my own amusement so you can argue it both ways. But, if I had been making money with that, I would have gone broke. All of this is a lot more subjective than it seems on the surface. It really isn't as simple as it looks and these assessments are more subjective than people want to believe. -- Paul Heinlein heinl...@madboa.com 45°22'48" N, 122°35'36" W
Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
On Mon, 27 Feb 2023, Ted Mittelstaedt wrote: The problem with that is that the assessment itself is biased. If a business owner is doing the assessment they tend to bias against cost. But, what happens if a customer calls at the very moment your receptionist's PC is crashed, and she says "sorry I can't help my computer is down" And that customer says "no problem" hangs up, calls someone else, then over the next decade develops $200k of business with that vendor? In this hypothetical exchange, a business owner who didn't realize that a single PC would make or break a significant customer relationship would in all likelihood not be in business for very long. If your core competencies depend on 100% uptime during business hours of all your computing systems, you're in a business that will sooner or later demand commensurate IT spending. Lost opportunity cost. It's not easy to quantify so the business owners doing the assessment on new gear tend to discount $downtimeRisk. Which is why So many small businesses remain small, to be perfectly frank. Personally as a 1 man shop I'm OK with remaining small. But if you are a small business owner who employs others, you have a responsibility to provide continued employment for them, and that means prioritizing $downtimeRisk. At least, that's my take on it. It's not that I disagree with your assessment, but I don't back off my initial opinion that most IT decisions are based on risk assessment, not technology assessement. The latter can inform but will rarely trump the former. To pursue the "I get to choose the hypothetical the proves my case" tack you took, consider the business owner who is cash poor but relatively time rich. Keeping cash on hand can justify the owner's need to spend extra time keeping a fragile set of systems working. I say "can," not "will" or "must," but I think the point is reasonable. -- Paul Heinlein heinl...@madboa.com 45°22'48" N, 122°35'36" W
Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
On Mon, 27 Feb 2023, Ted Mittelstaedt wrote: Lost opportunity cost. It's not easy to quantify so the business owners doing the assessment on new gear tend to discount $downtimeRisk. Which is why So many small businesses remain small, to be perfectly frank. Ted, It's actually called 'opportunity cost.' I guess that's becaue it saves typing five characters. :-) Rich
Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
The problem with that is that the assessment itself is biased. If a business owner is doing the assessment they tend to bias against cost. But, what happens if a customer calls at the very moment your receptionist's PC is crashed, and she says "sorry I can't help my computer is down" And that customer says "no problem" hangs up, calls someone else, then over the next decade develops $200k of business with that vendor? Lost opportunity cost. It's not easy to quantify so the business owners doing the assessment on new gear tend to discount $downtimeRisk. Which is why So many small businesses remain small, to be perfectly frank. Personally as a 1 man shop I'm OK with remaining small. But if you are a small business owner who employs others, you have a responsibility to provide continued employment for them, and that means prioritizing $downtimeRisk. At least, that's my take on it. Ted -Original Message- From: PLUG On Behalf Of Paul Heinlein Sent: Monday, February 27, 2023 8:39 AM To: Portland Linux/Unix Group Subject: Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown ) >IT systems, like every other business asset, are assessed primarily from a >risk-management POV, not a technological one. And, frankly, this is >>appropriate. Business owners need justifications for expenses. >If spending ($cheapGear + ($serviceCall * 3) + $downtimeRisk) is lower than >($bestGear), then the argument for the best gear is dicey.
Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
On Mon, 27 Feb 2023, Ted Mittelstaedt wrote: The Dell and HP systems require an extra license fee be paid to enable the remote tools and most of my customers are smaller. Their tendency is to try to press workstations into use as servers, it's a big stretch to get them to actually buy a real server like a Proliant, [...] Once I get them setup with real server hardware and they notice wow - the server isn't going down every week - then they start to become believers. [...] Caveat lector: I was Ted's editor long ago at the long-defunct Computer Bits magazine, and his articles became a key foundation for my jump from Windows to *nix systems. IT systems, like every other business asset, are assessed primarily from a risk-management POV, not a technological one. And, frankly, this is appropriate. Business owners need justifications for expenses. If spending ($cheapGear + ($serviceCall * 3) + $downtimeRisk) is lower than ($bestGear), then the argument for the best gear is dicey. That's not to say that the business owner's risk assessments may not change or even mature over time, but I've given up pretending I always understand a customer's overall risk profile. So Ted is perfectly right: the best gear fits a certain risk assessment. Perhaps an experienced consultant understands what the owner will only see at a later date. Perhaps. -- Paul Heinlein heinl...@madboa.com 45°22'48" N, 122°35'36" W
Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
Raised flooring went out with IBM servers, lol. Far easier to run overhead cable management. I'm not a fan of pulling a raised floor to get at a cable and finding a dead rat down there. The Dell and HP systems require an extra license fee be paid to enable the remote tools and most of my customers are smaller. Their tendency is to try to press workstations into use as servers, it's a big stretch to get them to actually buy a real server like a Proliant, let alone pay the additional fees to enable ILO. It's also kind of hard to reach those servers when the Internet connection itself is down. I have actually in a few cases in the past gotten 2-3 year old servers off Ebay for a particular customer who was resistant to the idea of paying real money to replace the typical 5 year old workstation box out of warranty under the CEO's desk that gets kicked occasionally. I've also supplied at very little cost (since I picked them up used for free or very little cost) relay racks and shelving and other accessories to some customers to outfit a closet as a "server room" Once I get them setup with real server hardware and they notice wow - the server isn't going down every week - then they start to become believers. But it takes a lot of baby steps and time for this. And there's a LOT of hack techs running around out there who are happy to continue nursing the 5 year old workstation boxes out of warranty under the CEO's desk that get kicked occasionally. I guess their MO is make money from service calls so they encourage that nonsense. I only do retainers so as I explain to my customers, _I_ have a financial incentive for things to NOT go down because if they are going down all the time, my retainer fee isn't going to cover my time, whereas if you are paying that fee and you never see me, then that's good for you because then things are never going down, got it? It's like a revelation to some of them. The other thing is that most smaller customers do not, in fact, have a real Terminal Server. What I do in those cases is either setup VPNs using Untangle as a firewall (Untangle has very slick support for OpenVPN) to replace the usual 4 port Netgear router or cablemodem/router combo, or I load Microsoft Remote Desktop Gateway Server on one of their servers than setup the RDP clients to use the GW server. They RDP into their desktops not a terminal server. The remote KVM's are cool but once more, you have to have an operating Internet connection for them to work. With my customers most of their downtime is due to workstation issues and Internet connectivity so a KVM is not going to help. And there's a whole circus to discuss on what is called "solid workstation hardware" It's why I only buy HP Elites and Pros nowadays for myself and my family members instead of the crappy stuff. But when a small business is looking at upgrading 10-20 desktops it's very hard for them to see why they should double their spend for good gear when they can get the cheap crap for half the cost of good gear. Ted -Original Message- From: PLUG On Behalf Of Robert Citek Sent: Sunday, February 26, 2023 10:19 PM To: Portland Linux/Unix Group Subject: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown ) On Sun, Feb 26, 2023 at 9:52 PM Ted Mittelstaedt wrote: > However during the entire pandemic I was still out and about - since > you can't do IT consulting on a server that's down remotely. By "server", I am assuming that you mean some system on rails in a rack in a datacenter with raised flooring, hot/cold aisles, redundant power/networking, and physical security. In that environment, you usually can ( and want to ) be able to work on a downed server remotely. For example, Dell has iDRAC/DRAC and HP has iLO. For those systems that don't have built-in out-of-band ( OOB ) management, there are multi-port KVM over IP switches with many having virtual USB/CDs and power control.[1] For single use, there is the Lantronix Spider which is also available with remote power control.[2] In other words, you can connect over the internet to the DRAC/KVM ( e.g. ssh ), upload an ISO of your OS onto the virtual CD, power cycle the box, and have full remote control from BIOS to RAID to OS repair/installation. If the issue is hardware, e.g. bad drive, bad power supply, you put in a service request to remote hands at the data center and have them hot-swap your cold spare for the bad device. You've given them a copy of your runbook. They know what to do. lf the system has truly failed, you have a new system sent to the data center. When it arrives, have remote hands swap the bad for the good, plugging it into the OOB so you can once again access it remotely. And they package and send the bad system back to wherever. On the other hand, if by "server" you mean
Re: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
On Sun, 26 Feb 2023, Robert Citek wrote: By "server", I am assuming that you mean some system on rails in a rack in a datacenter with raised flooring, hot/cold aisles, redundant power/networking, and physical security. In that environment, you usually can ( and want to ) be able to work on a downed server remotely. For example, Dell has iDRAC/DRAC and HP has iLO. For those systems that don't have built-in out-of-band ( OOB ) management, there are multi-port KVM over IP switches with many having virtual USB/CDs and power control.[1] For single use, there is the Lantronix Spider which is also available with remote power control.[2] In other words, you can connect over the internet to the DRAC/KVM ( e.g. ssh ), upload an ISO of your OS onto the virtual CD, power cycle the box, and have full remote control from BIOS to RAID to OS repair/installation. Robert, As a non-computer industry business owner I appreciate learning this. Not that I'll have need for it, but it deepens my understanding of business computer use. Thank you, Rich
[PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )
On Sun, Feb 26, 2023 at 9:52 PM Ted Mittelstaedt wrote: > However during the entire pandemic I was still out and about - since you > can't do IT consulting on a server that's down remotely. By "server", I am assuming that you mean some system on rails in a rack in a datacenter with raised flooring, hot/cold aisles, redundant power/networking, and physical security. In that environment, you usually can ( and want to ) be able to work on a downed server remotely. For example, Dell has iDRAC/DRAC and HP has iLO. For those systems that don't have built-in out-of-band ( OOB ) management, there are multi-port KVM over IP switches with many having virtual USB/CDs and power control.[1] For single use, there is the Lantronix Spider which is also available with remote power control.[2] In other words, you can connect over the internet to the DRAC/KVM ( e.g. ssh ), upload an ISO of your OS onto the virtual CD, power cycle the box, and have full remote control from BIOS to RAID to OS repair/installation. If the issue is hardware, e.g. bad drive, bad power supply, you put in a service request to remote hands at the data center and have them hot-swap your cold spare for the bad device. You've given them a copy of your runbook. They know what to do. lf the system has truly failed, you have a new system sent to the data center. When it arrives, have remote hands swap the bad for the good, plugging it into the OOB so you can once again access it remotely. And they package and send the bad system back to wherever. On the other hand, if by "server" you mean the five year old box that's out of warranty, sitting under the CEO's desk, and gets kicked every time they reach to answer the phone, then that's a different scenario. Although, attaching a Spider to it would be a nice option. [1] https://en.wikipedia.org/wiki/KVM_switch [2] https://www.lantronix.com/products/lantronix-spider/ Regards, - Robert
