Re: [systemd-devel] Journald stops logging when lots of output
On Thu, Sep 26, 2013 at 2:46 PM, Olav Vitters o...@vitters.nl wrote: On Wed, Sep 11, 2013 at 08:35:49PM +0200, Lennart Poettering wrote: (Of course, journald should not exit under any such circumstances, but to find that we first need to track down why it does that currently). Though it might hide problems, shouldn't journald be configured to automatically be restarted by systemd? I'm not sure but I believe it crashes three times in a row, which is how far systemd will go trying to revive it. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] User journal corrupted or uncleanly shut down, renaming and replacing
On Wed, Sep 11, 2013 at 6:33 PM, Lennart Poettering lenn...@poettering.net wrote: On Mon, 12.08.13 14:58, Pedro Francisco (pedrogfranci...@gmail.com) wrote: Hi, Systemd complains frequently about corrupted journal. Do note: I don't have the default mount options. $ dmesg |grep corrupte [ 51.766346] systemd-journald[181]: File /var/log/journal/06fde5edd4974fa9a343215f093f5aae/user-42.journal corrupted or uncleanly shut down, renaming and replacing. $ sudo rpm -qa systemd systemd-204-9.fc19.i686 $ cat /proc/mounts (...) /dev/sda6 / ext4 rw,seclabel,noatime,nobarrier 0 0 (...) In addition, I have an issue related to systemd-journald stopping logging after a while if there are lots of output, as I mentioned on http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/12373 titled Journald stops logging when lots of output . Do you think: a) mount options can affect systemd logging and b) these two issues (the one on this email about corrupted journal and the one on the linked thread about journald crashing when there are lots of output) may be related? Thanks in Advance, My guess is that for some reason the file systems are not properly unmounted during shutdown. Can you maybe try to get shutdown logs as described here? http://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1 Are you using LVM or something else weird? Are you using Dracut? Does the issue go away if you drop nobarrier from your mount options? Seems to be fixed on systemd-204-11.fc19.i686 (tried two boots, changed nothing else). If it happens again, I'll report back. Anyway, for the record: no LVM and it was default boot process for Fedora 19, so I guess I'm using dracut. Thank you for your time! -- Pedro ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] User journal corrupted or uncleanly shut down, renaming and replacing
Hi, Systemd complains frequently about corrupted journal. Do note: I don't have the default mount options. $ dmesg |grep corrupte [ 51.766346] systemd-journald[181]: File /var/log/journal/06fde5edd4974fa9a343215f093f5aae/user-42.journal corrupted or uncleanly shut down, renaming and replacing. $ sudo rpm -qa systemd systemd-204-9.fc19.i686 $ cat /proc/mounts (...) /dev/sda6 / ext4 rw,seclabel,noatime,nobarrier 0 0 (...) In addition, I have an issue related to systemd-journald stopping logging after a while if there are lots of output, as I mentioned on http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/12373 titled Journald stops logging when lots of output . Do you think: a) mount options can affect systemd logging and b) these two issues (the one on this email about corrupted journal and the one on the linked thread about journald crashing when there are lots of output) may be related? Thanks in Advance, -- Pedro ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald stops logging when lots of output
How should I see journald status/log? sudo systemctl status systemd-journald.service ? P.S.: /var/lib/systemd/coredump is empty. On Sun, Aug 4, 2013 at 5:15 AM, Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl wrote: On Sat, Aug 03, 2013 at 11:14:28PM +0100, Pedro Francisco wrote: (question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Some similar bugs appeared with jorunald apparently dying when the kernel logs a lot. [1] is recent, but I think I saw some other instances. journald should throttle kernel messages, and could ligitimately loose some when the kernel is really verbose, but of course it shouldn't exit. I would start by investigating why journald exited. Zbyszek [1] https://bugzilla.redhat.com/show_bug.cgi?id=990323 Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Journald stops logging when lots of output
(question migrated from t...@lists.fedoraproject.org ) On Fedora 19, journald ( systemd-204-9.fc19.i686 ) stops logging. I had enabled lots of iwl3945 debugging ( various messages for each iwl3945 interaction -- modprobe iwl3945 debug=0x47ff ) throughout the night and the last thing `journalctl -b` has is: Ago 03 02:25:45 s2 systemd-journal[29414]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29414]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=killed, status=10/USR1 Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29419]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29419]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. Ago 03 02:25:45 s2 systemd-journal[29423]: Allowing system journal files to grow to 1.4G. Ago 03 02:25:45 s2 systemd-journal[29423]: Journal started Ago 03 02:25:45 s2 systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE Ago 03 02:25:45 s2 systemd[1]: Started Trigger Flushing of Journal to Persistent Storage. It is now 10:35 and while dmesg has (lots) of new info, journalctl -f does not. Any tips on what may be wrong? -- Pedro ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Heavy IO -- need help on how to debug
Yesterday my system was slow and I checked iotop: systemd-journal was using 99% IO (but not CPU). I haven't restarted the machine, but kill systemd-journal didn't help, the process would respawn -- as expected -- and resume the heavy IO). I disabled storage (Storage=none) and restarted the service. It helped a lot, but I still get the ocasional slow down -- though a lot less severe. iotop output (idle prio is because I did a ionice -c3 on it): 23616 idle root3.14 M/s 14.86 K/s 0.00 % 99.62 % systemd-journald $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 15G 11G 4,1G 72% / System: up-to-date Fedora 17, x86 $ sudo rpm -qa systemd* systemd-sysv-44-21.fc17.i686 systemd-44-21.fc17.i686 systemd-analyze-44-21.fc17.i686 What else can I do to debug what is going on next time it happens? Thanks in Advance, -- Pedro Francisco ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
