This bug was fixed in the package logrotate - 3.8.7-2ubuntu2.16.10.1 --------------- logrotate (3.8.7-2ubuntu2.16.10.1) yakkety; urgency=medium
* createOutputFile: rename already existing file (LP: #1630516) - d/p/ubuntu/createOutputFile-eliminate-stat-open-TOCTOU-race.patch - d/p/ubuntu/createOutputFile-rename-already-existing-file.patch -- Christian Ehrhardt <christian.ehrha...@canonical.com> Wed, 22 Mar 2017 11:47:34 +0100 ** Changed in: logrotate (Ubuntu Yakkety) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Released Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Released Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron <root@warden> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root 56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root 24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron <root@trac> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r----- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r----- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r----- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r----- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r----- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r----- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r----- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r----- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r----- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } # system-specific logs may be configured here $ cat /etc/logrotate.d/rsyslog /var/log/syslog { rotate 7 daily missingok notifempty delaycompress compress postrotate reload rsyslog >/dev/null 2>&1 || true endscript } /var/log/mail.info /var/log/mail.warn /var/log/mail.err /var/log/mail.log /var/log/daemon.log /var/log/kern.log /var/log/auth.log /var/log/user.log /var/log/lpr.log /var/log/cron.log /var/log/debug /var/log/messages { rotate 4 weekly missingok notifempty compress delaycompress sharedscripts postrotate reload rsyslog >/dev/null 2>&1 || true endscript } manager@trac:/var/log$ cat /etc/logrotate.d/munin-node /var/log/munin/munin-node.log { daily missingok rotate 7 compress copytruncate notifempty create 640 root root } I am confident that is all the files which are relative to the configuration: manager@trac:/var/log$ rgrep syslog /etc/logrotate.* /etc/logrotate.conf:# use the syslog group by default, since this is the owning group /etc/logrotate.conf:# of /var/log/syslog. /etc/logrotate.conf:su root syslog /etc/logrotate.d/ufw: invoke-rc.d rsyslog reload >/dev/null 2>&1 || true /etc/logrotate.d/rsyslog:/var/log/syslog /etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true /etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true manager@trac:/var/log$ rgrep munin /etc/logrotate.* /etc/logrotate.d/munin-node:/var/log/munin/munin-node.log If there is anything else I can do to help diagnose the issue please let me know. Usual boiler plate bits: "trac" machine: $ lsb_release -rd Description: Ubuntu 14.04.5 LTS Release: 14.04 $ apt-cache policy logrotate logrotate: Installed: 3.8.7-1ubuntu1 Candidate: 3.8.7-1ubuntu1 Version table: *** 3.8.7-1ubuntu1 0 500 http://gb.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages 100 /var/lib/dpkg/status "warden" machine: $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ apt-cache policy logrotate logrotate: Installed: 3.8.7-2ubuntu2 Candidate: 3.8.7-2ubuntu2 Version table: *** 3.8.7-2ubuntu2 500 500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages 100 /var/lib/dpkg/status N: Ignoring file '50unattended-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: logrotate 3.8.7-1ubuntu1 ProcVersionSignature: Ubuntu 4.2.0-42.49~14.04.1-generic 4.2.8-ckt12 Uname: Linux 4.2.0-42-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.21 Architecture: amd64 Date: Wed Oct 5 11:14:53 2016 InstallationDate: Installed on 2015-12-14 (295 days ago) InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805) SourcePackage: logrotate UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/hundredpapercuts/+bug/1630516/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp