** Also affects: rsyslog (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1875493
Title:
[core] log rotation doesn't properly restart rsyslogd
Status in Snappy:
New
Status in rsyslog package in Ubuntu:
New
Bug description:
One of our commercial partners is developing a device agent snap for
UC16. One of their engineers just shared the following bug report:
I wanted to give you a heads up on an issue that we encountered with
one of our gateways (a Dell 3000) regarding a full disk (/writable).
It seems as though rsyslog had a bad logrotate config and was holding
on to deleted files as described here:
-
https://www.claudiokuenzler.com/blog/861/after-ubuntu-update-trusty-xenial-disk-filled-syslog-logrotate
- https://bugzilla.redhat.com/show_bug.cgi?id=1713358
Both df and du where showing different results for the “/writable”
path on the Dell gateway.
We were able to copy the lsof binary over and found this:
$ sudo /tmp/lsof | grep deleted
...
rsyslogd 1765 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
rsyslogd 1765 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
rsyslogd 1765 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
in:imuxso 1765 1776 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog
(deleted)
in:imuxso 1765 1776 syslog 6w REG 179,4 45056 130836 /var/log/auth.log
(deleted)
in:imuxso 1765 1776 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log
(deleted)
in:imklog 1765 1777 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog
(deleted)
in:imklog 1765 1777 syslog 6w REG 179,4 45056 130836 /var/log/auth.log
(deleted)
in:imklog 1765 1777 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log
(deleted)
rs:main 1765 1778 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog
(deleted)
rs:main 1765 1778 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
rs:main 1765 1778 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log
(deleted)
...
Restarting the service freed up the deleted files / disk space:
sudo service rsyslog restart
The rsyslog config (/etc/logrotate.d/rsyslog) calls this script post
rotate:
/usr/lib/rsyslog/rsyslog-rotate
Which does not actually kill the rsyslog process:
admin@GR0GP42:~$ ps aux | grep rsyslog
syslog 2305 0.7 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
admin 2464 0.0 0.0 12984 932 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
admin@GR0GP42:~$ sudo /usr/lib/rsyslog/rsyslog-rotate
admin@GR0GP42:~$ ps aux | grep rsyslog
syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
admin 2469 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
The fix appears to be using the following command: sudo systemctl
restart rsyslog >/dev/null 2>&1 || true
admin@GR0GP42:~$ ps aux | grep rsyslog
syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
admin 2482 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
admin@GR0GP42:~$ sudo systemctl restart rsyslog >/dev/null 2>&1 || true
admin@GR0GP42:~$ ps aux | grep rsyslog
syslog 2487 1.0 0.1 262692 3432 ? Ssl 20:36 0:00 /usr/sbin/rsyslogd -n
admin 2496 0.0 0.0 12984 980 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
To manage notifications about this bug go to:
https://bugs.launchpad.net/snappy/+bug/1875493/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
