[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
I have validated the trusty version works as expected! Thanks ** Tags added: verification-done-trusty -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r-
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Also validated on Yakkety. ** Tags added: verification-done-yakkety ** Tags removed: verification-needed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r--
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
This bug was fixed in the package logrotate - 3.8.7-1ubuntu1.1 --- logrotate (3.8.7-1ubuntu1.1) trusty; urgency=medium * createOutputFile: rename already existing file (LP: #1630516) - d/p/ubuntu/createOutputFile-eliminate-stat-open-TOCTOU-race.patch - d/p/ubuntu/createOutputFile-rename-already-existing-file.patch -- Christian Ehrhardt Wed, 22 Mar 2017 12:43:10 +0100 ** Changed in: logrotate (Ubuntu Trusty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Released Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Released Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
This bug was fixed in the package logrotate - 3.8.7-2ubuntu2.16.10.1 --- logrotate (3.8.7-2ubuntu2.16.10.1) yakkety; urgency=medium * createOutputFile: rename already existing file (LP: #1630516) - d/p/ubuntu/createOutputFile-eliminate-stat-open-TOCTOU-race.patch - d/p/ubuntu/createOutputFile-rename-already-existing-file.patch -- Christian Ehrhardt Wed, 22 Mar 2017 11:47:34 +0100 ** Changed in: logrotate (Ubuntu Yakkety) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Released Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Released Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Things are a bit more complex as it also changes a build time self test. These Tests are running, see https://launchpadlibrarian.net/228546427/buildlog_ubuntu-xenial-s390x.logrotate_3.8.7-2ubuntu2_BUILDING.txt.gz The changes to the tests add new tests for these cases, so I think we want that. And then there is a direct follow up fix for the fix which has to be considered as well. https://github.com/logrotate/logrotate/commit/aff4a30807218a52b6b5f200c5aa0eea335547ba OTOH T-Z are on almost the same version, so once done that should apply. Working on the backports for the SRU now. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop l
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
The second link is better as "https://launchpad.net/~ci-train-ppa- service/+archive/ubuntu/2624/" withut the tail +packages. (you get from one to the other on that page, so the initial link is not too bad) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } # system-specific logs may be configured here $ cat /e
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
The backports were as smooth as expected. I'm happy they added a test for the case upstream (and all the others that are there already). That gives me some confidence in the changes. I'll do some regression testing on my end and when all is complete we can start bringing it to zesty and from there consider SRUs. @tafthorne - Since you know the setup to trigger this particular issue I wanted to ask you if you could try the following ppa's and report back here. For Xenial, Yakkety and Zesty please use [1], and for Trusty [2]. - Also you could help me adding a proper SRU Template here, copy from [3] and modify the adapted version here in the top into the description. I can help you if needed, please catch me on IRC in that case. [1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2623 [2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2624/+packages [3]: https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man lo
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
SRU Bug Template [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] [Other Info] -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Moving the template up and adding more to the regression statement. Thanks a lot for the testcase! -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } # system-specific logs may be configured here $ cat /etc/logrotate.d/rsyslog /var/log/syslog { rotate 7 daily miss
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
** Description changed: - Good afternoon. - I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have + [Impact] + + Logrotate fails to rotate a log and then will continue to fail to rotate + it until manual intervention takes place. If messaging has not been + configured on the system there will be no warning issued to the user. + The log will grow day by day until a user intervenes or the drive that + the log is stored on is full. + + Very large log files can make it more difficult to find useful data. + Full drives make the rest of the system fail to operate. Backporting a + fix would prevent drives filling up on stable releases. + + [Test Case] + + Go to your logs area (/var/logs) and create a file with a name ending + .2, as would be created part way through the logrotate process. So if + you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, + /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your + subsequent log rotate runs will fail. + + [Regression Potential] + + - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). + - So far in those conditions it failed to rotate + - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. + - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. + - Therefore I consider the Potential low enough to consider the fix. + + [Other Info] + n/a + + --- + + Good afternoon. + I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 - When I inspected the area of concern I was able to see that there was an existing .1 file. + When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz - The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. + The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 - Different file, different machine but a very similar error message. - - Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. + Different file, different machine but a very similar error message. + + Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz T
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Installing for 16.04 xenial on a machine that has just reproduced the issue on its own. If I do not see the error message emailed out to me tonight then it is likely that the fix has succeeded. I shall try to post an update promptly, sorry for the delay. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
The proposed change added under https://launchpad.net/~ci-train-ppa- service/+archive/ubuntu/2623 seems to have resolved my issue. Before the change I received an email stating: /etc/cron.daily/logrotate: error: error creating output file /var/log/apt-cacher-ng/apt-cacher.err.1.gz: File exists error: error creating output file /var/log/auth.log.1.gz: File exists run-parts: /etc/cron.daily/logrotate exited with return code 1 After applying the change I have received an email stating: /etc/cron.daily/logrotate: error: destination /var/log/apt-cacher-ng/apt-cacher.err.1.gz already exists, renaming to /var/log/apt-cacher-ng/apt-cacher.err.1.gz-2017040406.backup I can also see what looks like a normal pattern of file names a modification dates: $ ls -l /var/log/apt-cacher-ng/ total 25772 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng0 Apr 4 06:26 apt-cacher.err -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 297 Mar 31 13:40 apt-cacher.err.1 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng0 Mar 31 06:26 apt-cacher.err.1.gz-2017040406.backup -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 29 06:25 apt-cacher.err.2.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 190 Mar 29 01:53 apt-cacher.err.3.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 27 06:25 apt-cacher.err.4.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 26 06:25 apt-cacher.err.5.gz ... Hopefully I will receive no email overnight tonight and log rotations will proceed as intended. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Thank you Thomas, the new mail content you referred to is already the fix in place and working. Thank you so much for double checking with me on these PPAs. So we have all in place: - SRU Template - Patches reviewed and prepared - Confidence in the fix by pre-testing ppas With that they are ready to be pushed those into the SRU process, which means they will get to the "unapproved" queue of the releases. The SRU Team will double check there if I had made any errors or if we both overlooked something. I hope that does not appear like a burden, the purpose of this process is to mimimize the risk to unintentionally regress users. But before that the next burden is that I can't upload logrotate on my own yet, so I have to subscribe ubuntu-sponsors to help us doing so. @Sponsors since this is tested via bileto tickets a publish on the two tickets might be the easiest way. https://bileto.ubuntu.com/#/ticket/2623 https://bileto.ubuntu.com/#/ticket/2624 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar err
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
To finish off my confirmation of the fix, I did not see any email notifications overnight. When I look at the relevant log files I can see the rotation is progressing: $ ls -l /var/log/apt-cacher-ng/apt-cacher.err* -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 0 Apr 5 06:28 /var/log/apt-cacher-ng/apt-cacher.err -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 0 Apr 4 06:26 /var/log/apt-cacher-ng/apt-cacher.err.1 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 0 Mar 31 06:26 /var/log/apt-cacher-ng/apt-cacher.err.1.gz-2017040406.backup -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 180 Mar 31 13:40 /var/log/apt-cacher-ng/apt-cacher.err.2.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 29 06:25 /var/log/apt-cacher-ng/apt-cacher.err.3.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 190 Mar 29 01:53 /var/log/apt-cacher-ng/apt-cacher.err.4.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 27 06:25 /var/log/apt-cacher-ng/apt-cacher.err.5.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 26 06:25 /var/log/apt-cacher-ng/apt-cacher.err.6.gz -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 20 Mar 25 06:25 /var/log/apt-cacher-ng/apt-cacher.err.7.gz > Thank you Thomas You are welcome. Thank you for fixing the bug (or porting the fix into Ubuntu). > I hope that does not appear like a burden, the purpose of this process is to mimimize the risk to unintentionally regress users. That is fine. I work in software development so I know the benefits of thorough QA. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it woul
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
X/Y/Z uploads are published. Due to the Freeze on the Zesty they are on unapproved queue there as well now. Once it migrated into Zesty-Release the SRU Team will start to consider it for the older releases. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
** Also affects: logrotate (Ubuntu Trusty) Importance: Undecided Status: New ** Also affects: logrotate (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: logrotate (Ubuntu Yakkety) Importance: Undecided Status: New ** Changed in: logrotate (Ubuntu Trusty) Status: New => In Progress ** Changed in: logrotate (Ubuntu Xenial) Status: New => In Progress ** Changed in: logrotate (Ubuntu Yakkety) Status: New => In Progress ** Changed in: logrotate (Ubuntu Zesty) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Committed Status in logrotate source package in Trusty: In Progress Status in logrotate source package in Xenial: In Progress Status in logrotate source package in Yakkety: In Progress Status in logrotate source package in Zesty: Fix Committed Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog*
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
This bug was fixed in the package logrotate - 3.8.7-2ubuntu3 --- logrotate (3.8.7-2ubuntu3) zesty; urgency=medium * createOutputFile: rename already existing file (LP: #1630516) - d/p/ubuntu/createOutputFile-eliminate-stat-open-TOCTOU-race.patch - d/p/ubuntu/createOutputFile-rename-already-existing-file.patch -- Christian Ehrhardt Wed, 22 Mar 2017 11:47:34 +0100 ** Changed in: logrotate (Ubuntu Zesty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: In Progress Status in logrotate source package in Xenial: In Progress Status in logrotate source package in Yakkety: In Progress Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
** Changed in: logrotate (Ubuntu Trusty) Importance: Undecided => High ** Changed in: logrotate (Ubuntu Xenial) Importance: Undecided => High ** Changed in: logrotate (Ubuntu Yakkety) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: In Progress Status in logrotate source package in Xenial: In Progress Status in logrotate source package in Yakkety: In Progress Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r-
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Hello Thomas, or anyone else affected, Accepted logrotate into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/logrotate/3.8.7-2ubuntu2.16.10.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: logrotate (Ubuntu Yakkety) Status: In Progress => Fix Committed ** Tags added: verification-needed ** Changed in: logrotate (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Committed Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
** Changed in: hundredpapercuts Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Fix Released Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Released Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Released Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
** Also affects: logrotate (Ubuntu Zesty) Importance: Critical Status: Triaged -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Triaged Status in logrotate source package in Zesty: Triaged Status in logrotate package in Debian: Fix Released Bug description: Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz -rw-r- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz -rw-r- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz -rw-r- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz There has been some speculation that a full or nearly full /var partition would cause this issue. I can confirm that /var is part of / on my systems and that presently both of them have several gigabytes of space. I run Munin an Icinga to monitor system state. Neither of these show / being completely full in the past month. They have both had /boot fill significantly. Trac had a highest use value of / being 99.28% full in the past year but warden has only had a peak of 33% in the past year. A quick search of the internet suggests a couple of other people reporting similar issues: https://github.com/gitlabhq/gitlabhq/issues/6894 http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about: $ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } # system-specific logs may be configured here $ cat /etc/logrotate.d/rsyslog /var/log/syslog { rotate 7 daily missingok
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Running through on Xenial. $ sudo apt-get install logrotate/xenial-proposed Reading package lists... Done Building dependency tree Reading state information... Done Selected version '3.8.7-2ubuntu2.16.04.1' (Ubuntu:16.04/xenial-proposed [amd64]) for 'logrotate' The following packages will be upgraded: logrotate 1 to upgrade, 0 to newly install, 0 to remove and 22 not to upgrade. Need to get 37.8 kB of archives. After this operation, 0 B of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 logrotate amd64 3.8.7-2ubuntu2.16.04.1 [37.8 kB] Fetched 37.8 kB in 0s (639 kB/s) (Reading database ... 889765 files and directories currently installed.) Preparing to unpack .../logrotate_3.8.7-2ubuntu2.16.04.1_amd64.deb ... Unpacking logrotate (3.8.7-2ubuntu2.16.04.1) over (3.8.7-2ubuntu2) ... Processing triggers for man-db (2.7.5-1) ... Setting up logrotate (3.8.7-2ubuntu2.16.04.1) ... $ cd /var/log/ $ sudo touch syslog.2 $ ls -l syslog* -rw-r- 1 syslog adm 193861 Apr 25 15:48 syslog -rw-r- 1 syslog adm 586698 Apr 25 07:35 syslog.1 -rw-r--r-- 1 root root 0 Apr 25 15:48 syslog.2 -rw-r- 1 syslog adm 26955 Apr 24 07:35 syslog.2.gz -rw-r- 1 syslog adm 26695 Apr 23 07:35 syslog.3.gz -rw-r- 1 syslog adm 31307 Apr 22 07:35 syslog.4.gz -rw-r- 1 syslog adm 32909 Apr 21 07:35 syslog.5.gz -rw-r- 1 syslog adm 39182 Apr 20 07:35 syslog.6.gz -rw-r- 1 syslog adm 27036 Apr 19 07:35 syslog.7.gz Either things will fail overnight (well first thing tomorrow morning or all will be well. I shall get back in touch soon. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Committed Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Looks to me like everything worked fine overnight. $ ls -l /var/log/syslog* -rw-r- 1 syslog adm 587351 Apr 26 12:10 /var/log/syslog -rw-r- 1 syslog adm 1824727 Apr 26 07:38 /var/log/syslog.1 -rw-r--r-- 1 root root 0 Apr 25 15:48 /var/log/syslog.2 -rw-r- 1 syslog adm37783 Apr 25 07:35 /var/log/syslog.2.gz -rw-r- 1 syslog adm26955 Apr 24 07:35 /var/log/syslog.3.gz -rw-r- 1 syslog adm26695 Apr 23 07:35 /var/log/syslog.4.gz -rw-r- 1 syslog adm31307 Apr 22 07:35 /var/log/syslog.5.gz -rw-r- 1 syslog adm32909 Apr 21 07:35 /var/log/syslog.6.gz -rw-r- 1 syslog adm39182 Apr 20 07:35 /var/log/syslog.7.gz The presence of the inconveniently named file looks to have been ignored and the log rotations continued unabated. > change the tag from verification-needed to verification-done I have only verified this on Xenia so far. I think that means I leave the tags alone for now right? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Committed Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Hi Thomas, thanks for your ongoing work. Look at [1] - in this case you'd add verification-done-xenial now. Once all needed tags are added (+Trusty, +Yakkety) you'd also remove the verification-needed. [1]: https://wiki.ubuntu.com/QATeam/PerformingSRUVerification#Updating_the_bug_report -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Committed Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
Hi Christian, A more careful read through of the instructions you link to and I have spotted the required tag. Now added that as requested. I'll try to get the other two tested using lsc tomorrow. Tag now added. ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Committed Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog adm 125819 Sep 30 06:25 syslog.1 -rw-r- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz -rw-r- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz -rw-r- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz -rw-r- 1 syslog adm
[Touch-packages] [Bug 1630516] Re: Logrotate doesn't clean old system logs, allowing them to fill the disk
This bug was fixed in the package logrotate - 3.8.7-2ubuntu2.16.04.1 --- logrotate (3.8.7-2ubuntu2.16.04.1) xenial; urgency=medium * createOutputFile: rename already existing file (LP: #1630516) - d/p/ubuntu/createOutputFile-eliminate-stat-open-TOCTOU-race.patch - d/p/ubuntu/createOutputFile-rename-already-existing-file.patch -- Christian Ehrhardt Wed, 22 Mar 2017 11:47:34 +0100 ** Changed in: logrotate (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to logrotate in Ubuntu. https://bugs.launchpad.net/bugs/1630516 Title: Logrotate doesn't clean old system logs, allowing them to fill the disk Status in One Hundred Papercuts: Triaged Status in logrotate package in Ubuntu: Fix Released Status in logrotate source package in Trusty: Fix Committed Status in logrotate source package in Xenial: Fix Released Status in logrotate source package in Yakkety: Fix Committed Status in logrotate source package in Zesty: Fix Released Status in logrotate package in Debian: Fix Released Bug description: [Impact] Logrotate fails to rotate a log and then will continue to fail to rotate it until manual intervention takes place. If messaging has not been configured on the system there will be no warning issued to the user. The log will grow day by day until a user intervenes or the drive that the log is stored on is full. Very large log files can make it more difficult to find useful data. Full drives make the rest of the system fail to operate. Backporting a fix would prevent drives filling up on stable releases. [Test Case] Go to your logs area (/var/logs) and create a file with a name ending .2, as would be created part way through the logrotate process. So if you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz, /var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your subsequent log rotate runs will fail. [Regression Potential] - I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way). - So far in those conditions it failed to rotate - Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names. - Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist. - Therefore I consider the Potential low enough to consider the fix. [Other Info] n/a --- Good afternoon. I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have $ logrotate --version logrotate 3.8.7 bundled in it. A few weeks ago I started getting root emails such as this: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/munin/munin-node.log.1: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 When I inspected the area of concern I was able to see that there was an existing .1 file. manager@warden:/var/log/munin$ ll total 580 drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./ drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../ -rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log -rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log -rw-r--r-- 1 root root56598 Sep 21 02:01 munin-node.log.1 -rw-r--r-- 1 root root24576 Aug 31 02:01 munin-node.log.2 -rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed. I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week: > Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) > > /etc/cron.daily/logrotate: > error: error creating output file /var/log/syslog.1.gz: File exists > run-parts: /etc/cron.daily/logrotate exited with return code 1 Different file, different machine but a very similar error message. Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file. manager@trac:/var/log$ ll syslog* -rw-r- 1 syslog adm 918492 Oct 5 10:30 syslog -rw-r- 1 syslog