Re: are unattended updates a good idea?
On 15-01-31 09:58:39, Ml Ml wrote: > i have got about 50 Debian 6+7 Servers. They are doing all kind of > things like Webserver, Mailserver, DNS, etc… > > I am using apticron to keep track of the updates, but i seem to use > more and more time updating the hosts. > > [...] > > Is anyone else facing the same problem? What are your experiences > doing (blind) automatic security updates. > > Or are you maybe using something completly diffrent like puppet? > > Whats your practical experience with lots of servers? (i am not > interested in theoretical advises :-P ) I've not using unattended upgrades or stuff like this, but apt-dater [1]: "apt-dater provides an ncurses frontend for managing package updates on a large number of remote hosts using SSH. It supports Debian-based managed hosts as well as rug (e.g. openSUSE) and yum (e.g. CentOS) based systems." It's a great tool, which I can fully recommend, using it to upgrade around 200 machines. A similar tool is UPDIAN, with which I've got no experience at all, and it's not packaged for Debian (yet?), but still, it may be worth a look. Cheers, Georg [1] https://packages.debian.org/wheezy/apt-dater [2] https://github.com/robhost/UPDIAN signature.asc Description: Digital signature
Re: are unattended updates a good idea?
On Sat, Jan 31, 2015 at 09:58:39AM +0100, Ml Ml wrote: > Is anyone else facing the same problem? What are your experiences > doing (blind) automatic security updates. I've done automatic updates for Debian under cfengine control for nine years and Ubuntu for perhaps one and a half. I started with a small number of systems and gradually expanded it out until it includes most servers. I can't think of any incidents where this has caused serious problems, but I still update a handful of systems exclusively by hand. The major problems, limitations and warnings I'm aware of are: - In /etc/apt/sources.list it's very important to specify the release by name: you should use e.g. 'wheezy' instead of 'stable'. Otherwise you risk breaking your systems when jessie is released. - It's important to have an easy way to turn off automatic updates on a particular system so that you can do maintenance work, and it's important to have an easy way to turn off automatic updates globally if you need to. - It's probably a good idea to have some way of delaying updates to your critical production systems until you've checked them out on your test and less-critical production systems. - My scripts do not reboot the system or restart any services that APT doesn't. This means I have to restart services or systems manually for library and kernel updates. I have Nagios check for systems that require a reboot and I restart them during a maintenance window. - In the past, certain upgrade failures have broken the kernel or bootloader badly enough to leave systems unbootable without manual attention. If memory serves, this was most common on squeeze or lenny but hasn't happened frequently on wheezy. It's important to review the transcript of the upgrade session before you reboot. - Starting last spring, certain unattended upgrades (possibly things that trigger man-db's maintainerscripts?) began to fail if /bin/sh is dash, but not if it's bash. My hunch is that this might be related to not having a controlling TTY attached when apt-get is run by cfagent. I don't have enough information at this point to be confident it's not related to the way I'm invoking it or to file a useful bug report against anything. I haven't worked on it recently, but I have notes saying it might be related to bug #439763. I use the following, but there are likely better ways to do it nowadays: - Before doing anything else: /usr/bin/apt-get update -qq --trivial-only - To install all available updates, security and otherwise: /bin/sh -c 'DEBIAN_FRONTEND=noninteractive DEBCONF_REDIR= /usr/bin/apt-get -o="DPkg::Options=--force-confdef,confold" upgrade -qq -y' ifelapsed=65 - To install a particular package (%s to be replaced with the name): /bin/sh -c 'DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get -o="DPkg::Options::=--force-confdef,confold" install -qq -y %s' - To remove a packages (%s to be replaced with the name): /bin/sh -c 'DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get remove -qq -y %s' On Sat, Jan 31, 2015 at 12:53:32PM +0100, Michael Zoet wrote: > You can do updates with Puppet (or every other configuration > management tool you like) but using it for updating the whole system > is not the way I would go. You would need to create a complete list > of installed packages on the server and keep this up2date in Puppet. This is more or less the approach I take for updating MacOS X with softwareupdate(8), as it has fewer packages and a worse history of updates breaking things than Debian and Ubuntu. -- William Aoki KD7YAFwa...@umnh.utah.edu5-1924 -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150201172924.ga13...@umnh.utah.edu
Re: are unattended updates a good idea?
On Sat, Jan 31, 2015 at 02:50:31PM +0100, Ml Ml wrote: > Thank you very much! Your comments has been really helpful. > > Cheers, > Mario > > On Sat, Jan 31, 2015 at 12:53 PM, Michael Zoet wrote: > > Hi, > >> > >> Hello List, > >> > >> i have got about 50 Debian 6+7 Servers. They are doing all kind of > >> things like Webserver, Mailserver, DNS, etc… > >> > >> I am using apticron to keep track of the updates, but i seem to use > >> more and more time updating the hosts. Also, you should note that some services might be restarted automatically during this process, so if you have long running nightly jobs or something similar it might cause some issues. This issue could also true in reverse; Some service might have to be manually restarted to load updated libraries and such. MySQL server upgrades might break nightly jobs because of restarts, kernel upgrades would probably need a reboot etc so you should keep an eye on these things. That being said: We have used unattended-upgrades on our servers for a couple of years and we have never had any problems with the packages themselves yet though, so this seems to be a smaller problem. Still, you should consider having a test server with tools like "needsrestart" and "apt-listchanges", and a test suite for your applications to check if they still work with the new packages and that every service is back to normal afterwards. Just sharing my thoughts about this. - Daniel -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150201051415.gb19...@noserver.visp.name
Re: are unattended updates a good idea?
Mario,
I use 'unattended-upgrades' on a couple hundred enduser desktop
workstations. The idea being that most potential exploits in our
environment might be through end-user browser/surfing.
I choose not to use it on a few hundred servers, most of which are internal
or perform specialized scientific work, at this point in time. We have a
very limited set of servers exposed to the Internet and we tend to manage
them manually. (see below for 'pdsh' and approach to servers)
I also have 'apper' installed with a change to the policy-kit rules to
allow ANYONE (at the console) to *update* packages, but not to install new
packages w/o root. (dual-approach to updates, convenient if i ever have to
backout unattended-upgrades for some reason)
I enable the update-notifier-kde package as well (kingston-notifier?)
because 'apper's notifier popup doesn't reliably notify users of
reboot-required events, even though it is supposed to
('update-notifier-kde' was supposedly deprecated for this reason).
update-notifier-kde does trigger occasional popups successfully.
The hope here is that the end users will actually reboot the systems from
being prompted by update-notifier-kde when an automatic security patch
requiring reboot occurs. I have limited success with that. Probable
reasons are:
- users are not easy to train -- we also have lots of transient visitors.
- users do not want to interrupt workflow
- users often run multi-day/week/month long scientific jobs
- users may just not "get" that they really should reboot
- Wheezy has been much worse than Squeeze for updates that trigger a
required reboot. Users may be getting "burnt-out" on the frequency.
while not strictly addressing your question w.r.t. "servers", my experience
with unattended-upgrades on the workstations over the past 3 years has been
good. I setup e-mails to myself -- so i have been notified a few times
when a configuration file change blocks updates.
I have also setup 'apt-listbugs' on my systems, and 'unattended-upgrades'
will sometimes block hard, because it has an issue where 'apt-listbugs'
detects that ONE (or more) packages has a problem and returns a failcode
causing UA to abort the upgrade entirely (no piecemeal re-attempt). This
requires manual intervention. I have upped the threshhold on
'apt-listbugs' to critical to avoid that condition recurring frequently.
My experience in the past 10 years with Debian, is that i have maybe seen
ONE update that had some fallout that was at all inconvenient.
For my servers, i will typically use 'pdsh' (LLNL distributed shell) to
perform a global run of a forced updates. ('pdsh' is very helpful here).
I use both 'dsh-group's for personal use, and an nfs-shared
/usr/local/etc/genders (libgenders) file which contains attributes like
'os=debian_linux' to select servers/systems to hit
so, something like (genders module in play):
pdsh -lroot -g 'os=debian_linux&&hwtype=server' 'aptitude update -q=2 &&
aptitude safe-upgrade -q=2 --assume-yes ... {pkgs...}' 2>&1 | tee
upgrade.log
(it's important to use safe-upgrade, otherwise if you force-yes with
noninteractive in batch mode like this, aptitude can end up removing
packages if a package you specify doesn't have explicit dependency packages
also listed on the command line. It's always a good idea to run this
command on one server first to make sure it doesn't end up doing something
you didn't mean to. said from experience :-( )
(i'm at home and don't have the full invocations of Dpkg options for stuff
like retaining old configfiles and using noninteractive DPKG UI (via
env-var -- i can get that later, if you're interested)
--stephen
On Sat, Jan 31, 2015 at 1:58 AM, Ml Ml wrote:
> Hello List,
>
> i have got about 50 Debian 6+7 Servers. They are doing all kind of
> things like Webserver, Mailserver, DNS, etc…
>
> I am using apticron to keep track of the updates, but i seem to use
> more and more time updating the hosts.
>
> Recently i came across the unattended-upgrade project
> https://wiki.debian.org/UnattendedUpgrades.
>
> Do you think it is a good idea to do security updates automatically? I
> just don’t want to wake up one morning not having ssh access to my
> Servers because an update broke everything. The servers are still very
> important. I should not crash them at any time. On the other hand i
> would like to be up2date with my security patches.
>
> Is anyone else facing the same problem? What are your experiences
> doing (blind) automatic security updates.
>
> Or are you maybe using something completly diffrent like puppet?
>
> Whats your practical experience with lots of servers? (i am not
> interested in theoretical advises :-P )
>
> Thanks a lot,
> Mario
>
>
> --
> To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmas...@lists.debian.org
> Archive:
> https://lists.debian.org/canfxojb-wbmsn3q4i4eqevvdnjbsg5wrskhdvrybebzgw5...@mail
Re: are unattended updates a good idea?
Thank you very much! Your comments has been really helpful. Cheers, Mario On Sat, Jan 31, 2015 at 12:53 PM, Michael Zoet wrote: > Hi, >> >> Hello List, >> >> i have got about 50 Debian 6+7 Servers. They are doing all kind of >> things like Webserver, Mailserver, DNS, etc… >> >> I am using apticron to keep track of the updates, but i seem to use >> more and more time updating the hosts. > > > I use apticron, cron-apt on various servers for several years now and never > had an issue with them. >> >> >> Recently i came across the unattended-upgrade project >> https://wiki.debian.org/UnattendedUpgrades. >> >> Do you think it is a good idea to do security updates automatically? > > > I use unattended upgrades so far only for one server for some month. Never > had an issue with it. But for me there is not much difference in using > apticron, cron-apt or unattended upgrades mechanism. > >> I >> just don’t want to wake up one morning not having ssh access to my >> Servers because an update broke everything. The servers are still very >> important. I should not crash them at any time. On the other hand i >> would like to be up2date with my security patches. > > > Normally these tools only install security updates in a safe way. Meaning > they should not do a major version upgrading of any installed software. So > breaking something is most unlikely but no one can guarantee that. That's > why you should always have a plan b, regardless what you setup. How this is > setup depends heavily on your network layout and what kind of hardware or > virtualization is used. >> >> >> Is anyone else facing the same problem? What are your experiences >> doing (blind) automatic security updates. >> >> Or are you maybe using something completly diffrent like puppet? > > > You can do updates with Puppet (or every other configuration management tool > you like) but using it for updating the whole system is not the way I would > go. You would need to create a complete list of installed packages on the > server and keep this up2date in Puppet. This only moves the problem to > Puppet... And then you might have different package base on different > servers. This needs also be tracked. Other tools (like the mentioned 3) are > better for this. But you should use Puppet (or every other configuration > tool) to setup an automatic security update mechanism. > >> >> Whats your practical experience with lots of servers? (i am not >> interested in theoretical advises :-P ) >> > > If you have "lots" (for some this means 1000 of servers, for others 10 is > already a lot...) of servers you should use a configuration management tool > that automatically sets up automatic security updates. The mentioned tools > already provide you with everything you need on a Debian system. What you > use is a matter of taste. > In the past years I have setup this mechanism on about 400 servers and never > had real big issues. Sometimes the package list updates are stuck but mostly > recover in the next try. And if something is really wrong you can always > login to the server and repair the problem manually. Monitoring these kind > of things is really important but is a completely different topic. > > Michael > > > > > > > -- > To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org > with a subject of "unsubscribe". Trouble? Contact > listmas...@lists.debian.org > Archive: https://lists.debian.org/54ccc23c.5020...@zoet.de > -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/canfxojdvxjht8q28hujybwbs6r_e43onwqroyevwyahwf8w...@mail.gmail.com
Re: are unattended updates a good idea?
Hi, Hello List, i have got about 50 Debian 6+7 Servers. They are doing all kind of things like Webserver, Mailserver, DNS, etc… I am using apticron to keep track of the updates, but i seem to use more and more time updating the hosts. I use apticron, cron-apt on various servers for several years now and never had an issue with them. Recently i came across the unattended-upgrade project https://wiki.debian.org/UnattendedUpgrades. Do you think it is a good idea to do security updates automatically? I use unattended upgrades so far only for one server for some month. Never had an issue with it. But for me there is not much difference in using apticron, cron-apt or unattended upgrades mechanism. I just don’t want to wake up one morning not having ssh access to my Servers because an update broke everything. The servers are still very important. I should not crash them at any time. On the other hand i would like to be up2date with my security patches. Normally these tools only install security updates in a safe way. Meaning they should not do a major version upgrading of any installed software. So breaking something is most unlikely but no one can guarantee that. That's why you should always have a plan b, regardless what you setup. How this is setup depends heavily on your network layout and what kind of hardware or virtualization is used. Is anyone else facing the same problem? What are your experiences doing (blind) automatic security updates. Or are you maybe using something completly diffrent like puppet? You can do updates with Puppet (or every other configuration management tool you like) but using it for updating the whole system is not the way I would go. You would need to create a complete list of installed packages on the server and keep this up2date in Puppet. This only moves the problem to Puppet... And then you might have different package base on different servers. This needs also be tracked. Other tools (like the mentioned 3) are better for this. But you should use Puppet (or every other configuration tool) to setup an automatic security update mechanism. Whats your practical experience with lots of servers? (i am not interested in theoretical advises :-P ) If you have "lots" (for some this means 1000 of servers, for others 10 is already a lot...) of servers you should use a configuration management tool that automatically sets up automatic security updates. The mentioned tools already provide you with everything you need on a Debian system. What you use is a matter of taste. In the past years I have setup this mechanism on about 400 servers and never had real big issues. Sometimes the package list updates are stuck but mostly recover in the next try. And if something is really wrong you can always login to the server and repair the problem manually. Monitoring these kind of things is really important but is a completely different topic. Michael -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/54ccc23c.5020...@zoet.de
Re: are unattended updates a good idea?
We use cron-apt for over a year now to patch around 120 Debian Servers with security fixes every night. In this time we never had a broken security update. But we mostly use them as Webservers or Appservers who run Java-Apps. So if u use highly specialized Software you need to consider for yourself. Maybe run it on test setup first. Charm is that critical patches like Bash were patched in less than 24 hours. Hope that helps. Greetings MattiasAm 31.01.2015 10:45 schrieb Andrew Beverley : > > On Sat, 2015-01-31 at 09:58 +0100, Ml Ml wrote: > > Do you think it is a good idea to do security updates automatically? > > I've always avoided this for the same reasons as you, but thinking back > over the last 10 years, I don't think I've ever had an update break > something, so maybe it's time to try... > > > Or are you maybe using something completly diffrent like puppet? > > I currently use Ansible, which I run weekly against all my servers, > unless there is a critical update in which case I run it immediately. > For a variety of reasons I'm about to move to Rexify, but the concept > will be the same. > > > Whats your practical experience with lots of servers? (i am not > > interested in theoretical advises :-P ) > > I'd be interested to hear other people's ideas and experiences though. > > Andy > > > > -- > To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org > with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org > Archive: https://lists.debian.org/1422697558.21256.2.camel@andy-laptop >
Re: are unattended updates a good idea?
On Sat, 2015-01-31 at 09:58 +0100, Ml Ml wrote: > Do you think it is a good idea to do security updates automatically? I've always avoided this for the same reasons as you, but thinking back over the last 10 years, I don't think I've ever had an update break something, so maybe it's time to try... > Or are you maybe using something completly diffrent like puppet? I currently use Ansible, which I run weekly against all my servers, unless there is a critical update in which case I run it immediately. For a variety of reasons I'm about to move to Rexify, but the concept will be the same. > Whats your practical experience with lots of servers? (i am not > interested in theoretical advises :-P ) I'd be interested to hear other people's ideas and experiences though. Andy -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/1422697558.21256.2.camel@andy-laptop
are unattended updates a good idea?
Hello List, i have got about 50 Debian 6+7 Servers. They are doing all kind of things like Webserver, Mailserver, DNS, etc… I am using apticron to keep track of the updates, but i seem to use more and more time updating the hosts. Recently i came across the unattended-upgrade project https://wiki.debian.org/UnattendedUpgrades. Do you think it is a good idea to do security updates automatically? I just don’t want to wake up one morning not having ssh access to my Servers because an update broke everything. The servers are still very important. I should not crash them at any time. On the other hand i would like to be up2date with my security patches. Is anyone else facing the same problem? What are your experiences doing (blind) automatic security updates. Or are you maybe using something completly diffrent like puppet? Whats your practical experience with lots of servers? (i am not interested in theoretical advises :-P ) Thanks a lot, Mario -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CANFxOjB-WBmSn3Q4i4eqEvvDNjbSG5wRSkHdvrYBeBzGW=5...@mail.gmail.com
