Re: Bug in yum-autoupdate
On Sat, Aug 03, 2013 at 12:59:55PM -0400, Nico Kadel-Garcia wrote: ... I've helped run, and done OS specifications and installers for a system over 10,000 hosts. and you *never*, *never*, *never* auto-update them without ... Okey, now I get it. I run 50 hosts, you run 10,000 hosts, the defaults should be set your way. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: Bug in yum-autoupdate
Here here-- Sent from my HP Pre3On Aug 4, 2013 15:13, Mark Stodola stod...@pelletron.com wrote: Can we please end this discussion? It has become extremely off topic at this point and no longer has _any_ direct impact on Scientific Linux.
Re: Bug in yum-autoupdate
On Thu, Aug 1, 2013 at 6:07 PM, Steven Haigh net...@crc.id.au wrote: On 02/08/13 02:26, Vincent Liggio wrote: On 08/01/2013 12:16 PM, Elias Persson wrote: All the more reason to read up on the differences, and if it's only one system 'yum remove yum-autoupdate' is hardly a big deal. If it's 1200 systems, what difference would an option in anaconda make? It's not like you'll be stepping through that hundreds of times, right? No, when I have to migrate to a new OS (which won't be a 6.4 derivative, it will be a 7.0 one, so probably 8-9 months from now), then I'll worry about the differences. When I'm testing a piece of hardware that requires a specific kernel release on an OS I don't run, whether a new option is installed by default or not is not on the top of my list of things to worry about. If you really do have 1200 systems to worry about, I'd be looking at things like satellite. I have ~20-25 systems and yum-autoupdate is fantastic. It does what it says on the box and relieves me of having to watch / check for updates every day. I get an email in the morning that tells me what was updated and if there were any problems. It's exceedingly dangerous in a production environment. I've helped run, and done OS specifications and installers for a system over 10,000 hosts. and you *never*, *never*, *never* auto-update them without warning or outside the maintenance windows. *Never*. If I caught someone else on the team doing that as a matter of policy, I would have campaigned to have them fired ASAP. A simple yum check-update cron job reporting to root or to a designated email address is far, far, far safer. Safer yet, if you have Nagios or Icinga up and running for production, is to install and use the nagios-plugins-check-updates package from EPEL, which allows graceful remote Nagios monitoring of the system status in an organized fashion across the network. In a production environment, unannounced or unplanned restarts of critical daemons such as httpd, mysql, named, snmpd, nagios, mrtg, or especially the Java based services such as tomcat6 can cause cascading failures across the whole environment. You may have the spare resources to do monkeywrench failures across your environment all the time to try to avoid this sort of thing, but very few facilities do. It's also nasty when you have software that is incompatible with contemporary versions of upstream published software. Take a look over at https://github.com/opentusk/OpenTUSK-SRPMS for some work I did last year. Some of the components listed there were more modern than those in SL 6, but many of those whose names start with tusk-, were built to avoid the automatic updates to contemporary release of of that software which was incompatible with the existing codebase. And some software updates, such as database updates, are *not reversible* without enormous engineering pain. When SL 5 updated from subversion-1.4.2 to subversion-1.6.11, it auto-updated the local Subversion repositories the next time you opened them, and *they can't be turned back!!!* and are incompatible with older versions of Subversion.
Re: Bug in yum-autoupdate
- Original Message - From: Nico Kadel-Garcia nka...@gmail.com It's exceedingly dangerous in a production environment. I've helped run, and done OS specifications and installers for a system over 10,000 hosts. and you *never*, *never*, *never* auto-update them without warning or outside the maintenance windows. *Never*. If I caught someone else on the team doing that as a matter of policy, I would have campaigned to have them fired ASAP. If you have to manage 10,000 hosts then you are lucky you never had to learn to deal with no maintenance window and 0 downtime, and so most of your maintenance had to be possible outside of a maintenance window. That is how many IT shops with thousands of machines have to operate these days. You might even want to read up on Netflix's thoughts on chaos monkey. Autoupgrades are just another form of random outage you might have to deal with. As long as you have different hosts upgrading on different days and times, and you have automated routines that test and take servers out of service automatically if things fail, then autogrades is perfectly fine. If things break from the autoupgrades, it becomes real obvious based on the update history of which machines broke from it. Campaigning to have someone fired without even hearing their reason for upgrading, or even warning them first that at your location is is standard practice not to ever autoupgrade because you have a separate QA process that even critical security patches must go through is a very bad practice on your part. I am not going to state what patch policy I use, only that different policies work for different environments. Based on your statement, it sounds like you could be loosing some valuable co-workers by lobbying to get people fired that have a different opinion from you instead of trying to educate and/or learn from each other. If you feel you can not learn from your peers, you have already proven you are correct in that respect, but you have also shown there is much you don't know by being incapable of learning new things. (Personally I would hate to use Nagios for 10,000 hosts. It didn't really scale that well IMHO, but to be honest I haven't bothered looking at it in over 4 years, and maybe it's improved. Not familiar with Icinga, but I have had good luck with Zabbix for large scale)
Re: Bug in yum-autoupdate
On Sat, Aug 3, 2013 at 9:28 PM, John Lauro john.la...@covenanteyes.com wrote: - Original Message - From: Nico Kadel-Garcia nka...@gmail.com It's exceedingly dangerous in a production environment. I've helped run, and done OS specifications and installers for a system over 10,000 hosts. and you *never*, *never*, *never* auto-update them without warning or outside the maintenance windows. *Never*. If I caught someone else on the team doing that as a matter of policy, I would have campaigned to have them fired ASAP. If you have to manage 10,000 hosts then you are lucky you never had to learn to deal with no maintenance window and 0 downtime, and so most of your maintenance had to be possible outside of a maintenance window. That is how many IT shops with thousands of machines have to operate No, you schedule the updates. A maintenance window is not the same as scheduled downtime, and for larger environments, you can schedule them for a planned set of well defined updates. For example, before allowing the system wide changes, you test them in a lab with a variety of the services and hardware you use in the field. And you don't test whatever the upstream vendor happened to publish lately, sight unseen, plus whatever they added between the test and the permitted update. You set up a defined set of updates, such as using a yum mirror snapshot (for Scientific Linux or CentOS) or a well defined RHN configuration. these days. You might even want to read up on Netflix's thoughts on chaos monkey. I'm familiar with the concept, and it has uses. However, having a chaos monkey in place does not reduce the risk of *network wide* auto-updates of corrupting all operating system configurations. I'm afraid I've had that happen: A kernel update introduced a regression, and took down over 1000 systems all the same night. (A vendor had changed hardware without notifying us, and the new kernel didn't have the right drivers for it: the old kernel did). Fortunately, it happened at a well defined maintenance window. And also fortunately, I'd taken advantage of the old LILO default and boot once oncly with a different setting tools to boot new kernels in test mode, and retain the old kernel as default after a power cycle if the new kernel failed to boot. Autoupgrades are just another form of random outage you might have to deal with. As long as And you deal with them by *turning them off* and scheduling the updates, with the chance to assess the updates. That's because leaving them enabled by default is not random. It's scheduled arbitrarily by the upstream vendor, and even *they* publish release notes and provide their own entire system (RHN, or spacewalk if you use the free versions) to schedule them. you have different hosts upgrading on different days and times, and you have automated routines that test and take servers out of service automatically if things fail, then autogrades is perfectly fine. If things break from the autoupgrades, it becomes real obvious based on the update history of which machines broke from it. Gee, you mean that you don't let systems automatically update without planning, and update different members at different scheduled and times? Why didn't I think of something like that? You must be smart! Campaigning to have someone fired without even hearing their reason for upgrading, or even warning them first that at your location is is standard practice not to ever autoupgrade because you have a separate QA process that even critical security patches must go through is a very bad practice on your part. Oh, he or she would get a chance to talk. If they spouted the auto-updating is safe mantra and refused to budge, I'd be like them like white on rice. Touching production servers unannounced is a serious no-no in large networks or any large network. I am not going to state what patch policy I use, only that different policies work for different environments. Based on your statement, it sounds like you could be loosing some valuable co-workers by lobbying to get people fired that have a different opinion from you instead of trying to educate and/or learn from each other. If you feel you can not learn from your peers, you have already proven you are correct in that respect, but you have also shown there is much you don't know by being incapable of learning new things. Oh, if they're *trainable*, they might get a shot. But leaving out the schedule the updates so they don't all occur at once part, as you did at first, is pretty dangerous. (Personally I would hate to use Nagios for 10,000 hosts. It didn't really scale that well IMHO, but to be honest I haven't bothered looking at it in over 4 years, and maybe it's improved. Not familiar with Icinga, but I have had good luck with Zabbix for large scale) Oh, you split it for a network that big It handles a thousand hosts reasonably well, even 10 years ago, if you don't go
Re: Bug in yum-autoupdate
On 02/08/13 02:26, Vincent Liggio wrote: On 08/01/2013 12:16 PM, Elias Persson wrote: All the more reason to read up on the differences, and if it's only one system 'yum remove yum-autoupdate' is hardly a big deal. If it's 1200 systems, what difference would an option in anaconda make? It's not like you'll be stepping through that hundreds of times, right? No, when I have to migrate to a new OS (which won't be a 6.4 derivative, it will be a 7.0 one, so probably 8-9 months from now), then I'll worry about the differences. When I'm testing a piece of hardware that requires a specific kernel release on an OS I don't run, whether a new option is installed by default or not is not on the top of my list of things to worry about. If you really do have 1200 systems to worry about, I'd be looking at things like satellite. I have ~20-25 systems and yum-autoupdate is fantastic. It does what it says on the box and relieves me of having to watch / check for updates every day. I get an email in the morning that tells me what was updated and if there were any problems. I've been doing this for several years with no problems. Before yum-autoupdate I had my own script do similar things in the daily cron. My point is, what you want (the issue being highlighted) is already being done. It's not being done precisely where you want it to be done, but I don't see how that's an issue, given the circumstances. What I think should be done is it be an obvious option, not hidden in release notes. Its hardly hidden - and if you don't like it, don't install the package - its purely in your control.
Re: Bug in yum-autoupdate
Seriously are we still beating this dead horse. While I admit I was the one who took this conversation on a tangent in the first place, every valid point of view on this has been covered from both sides.No resolution will come of it!From here its an intellectual pissing contest lets end it!-- Sent from my HP Pre3On Aug 1, 2013 20:08, Steven Haigh net...@crc.id.au wrote: On 02/08/13 09:59, Vincent Liggio wrote: On 08/01/2013 06:07 PM, Steven Haigh wrote: If you really do have 1200 systems to worry about, I'd be looking at things like satellite. I have ~20-25 systems and yum-autoupdate is fantastic. It does what it says on the box and relieves me of having to watch / check for updates every day. I get an email in the morning that tells me what was updated and if there were any problems. Guess none of you have to deal with third party applications, device drivers, change management, etc. Simple servers are easy to patch, and yes, I've done that for years. But take a system running anything graphical (especially with video and audio device drivers) and try to randomly patch it, and see how long that lasts! I hate to say it, but now you've shifted the goal posts. You talk about blade servers, now you talk about graphics drivers and audio - which I assume would be desktop use. Even on the desktop though, the kernel doesn't auto-update - so any graphics drivers that are installed against a specific kernel version will continue to work until you upgrade the kernel manually - at which time you will be required to build the kernel modules again (nvidia / ATI etc). (and yes, I really do have 1200+ systems to worry about. And I sleep very happily knowing tomorrow they won't be any different than they were today) Unless in the lack of updates, you leave a security hole and due to the lack of updates you never pick up on it. My 16 years of experience says that this is a dangerous attitude for system admins to adopt. And no, in 16 years I have never had a security breach (touch wood). Its hardly hidden - and if you don't like it, don't install the package - its purely in your control. It installs by default. I certainly can uninstall it, or set it to not autoupdate, which I shall. And this may work for you - and thats great for you. It shouldn't however mean that the default should be changed to disable this in the entire distro. In fact, if you *really* want to disable auto-updates globally, then you're better off using a single line sed command that you can run via SSH to all systems you control to disable it. That way it is rapidly deployed to all your systems with a simple bash script loop.
Bug in yum-autoupdate
I just installed SL 6.4 on a test blade and magically last night yum-autoupdate ran and updated bind. The default for yum-autoupdate in /etc/sysconfig/yum-autoupdate is supposed to be ENABLED=false but the actual value is set to true by default. Vince
RE: Bug in yum-autoupdate
Pat can comment more specifically but for 6.4 I believe it was changed to true by default. - Chris -Original Message- From: owner-scientific-linux-us...@listserv.fnal.gov [mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Vincent Liggio Sent: Wednesday, July 31, 2013 9:36 AM To: SCIENTIFIC-LINUX-USERS@FNAL.GOV Subject: Bug in yum-autoupdate I just installed SL 6.4 on a test blade and magically last night yum-autoupdate ran and updated bind. The default for yum-autoupdate in /etc/sysconfig/yum-autoupdate is supposed to be ENABLED=false but the actual value is set to true by default. Vince
Re: Bug in yum-autoupdate
It shows default of false still, and I think in general anything like this should not be automatic without the conscious decision of the user. On 07/31/2013 10:39 AM, Brown, Chris (GE Healthcare) wrote: Pat can comment more specifically but for 6.4 I believe it was changed to true by default. - Chris -Original Message- From: owner-scientific-linux-us...@listserv.fnal.gov [mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Vincent Liggio Sent: Wednesday, July 31, 2013 9:36 AM To: SCIENTIFIC-LINUX-USERS@FNAL.GOV Subject: Bug in yum-autoupdate I just installed SL 6.4 on a test blade and magically last night yum-autoupdate ran and updated bind. The default for yum-autoupdate in /etc/sysconfig/yum-autoupdate is supposed to be ENABLED=false but the actual value is set to true by default. Vince
Re: Bug in yum-autoupdate
On Wed, Jul 31, 2013 at 10:36:18AM -0400, Vincent Liggio wrote: I just installed SL 6.4 on a test blade and magically last night yum-autoupdate ran and updated bind. The default for yum-autoupdate in /etc/sysconfig/yum-autoupdate is supposed to be ENABLED=false but the actual value is set to true by default. Where is the bug? Automatic updates have always been enabled by default for as long as I remember. Today is 2013 not 1993, the Internet is not a gentlemen's club anymore and keeping up with updates on Internet-connected devices is no longer optional. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
