Re: [systemd-devel] logging in RAM and journald configuration issue
On Di, 02.04.19 17:42, Dave Howorth (syst...@howorth.org.uk) wrote: > On 2019-03-20 Lennart Poettering wrote: > > > On Mi, 20.03.19 18:08, Dave Howorth (syst...@howorth.org.uk) wrote: > > > > > > > > At present AFAICT the log2ram.service runs > > > Before=systemd-journald.service and various other services, so I > > > think that aspect is covered. > > > > I am not sure what this service does, but if all you want to do is > > make changes to /var/log/journal before journald writes there it > > should be sufficient to order it before systemd-journal-flush.service > > which is nowadays the clear boundary when journald starts writing to > > /var. > > Sorry for the late reply, but I'm just trying to understand this > better. Is systemd-journal-flush.service documented anywhere? It just invokes "journalctl --flush" which is documented here: https://www.freedesktop.org/software/systemd/man/journalctl.html#--flush Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] logging in RAM and journald configuration issue
On 2019-03-20 Lennart Poettering wrote: > On Mi, 20.03.19 18:08, Dave Howorth (syst...@howorth.org.uk) wrote: > > > > > At present AFAICT the log2ram.service runs > > Before=systemd-journald.service and various other services, so I > > think that aspect is covered. > > I am not sure what this service does, but if all you want to do is > make changes to /var/log/journal before journald writes there it > should be sufficient to order it before systemd-journal-flush.service > which is nowadays the clear boundary when journald starts writing to > /var. Sorry for the late reply, but I'm just trying to understand this better. Is systemd-journal-flush.service documented anywhere? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] logging in RAM and journald configuration issue
Back then, this is something we have tried and it works. Not with bind mount though. We also have SD-card as primary storage. As soon as the storage service mounted the SD-Card, we have created a symbolink link under /var/log to the persistent and triggered journald flush. It worked just fine! On Wed, Mar 20, 2019 at 4:54 PM Colin Guthrie wrote: > > Lennart Poettering wrote on 20/03/2019 13:37: > > On Mi, 20.03.19 09:57, Dave Howorth (syst...@howorth.org.uk) wrote: > > > >>> On Di, 19.03.19 20:45, Dave Howorth (syst...@howorth.org.uk) wrote: > >>> > In the case of a machine that uses an SD card as its primary backing > store, it is desirable to reduce the number of write operations to > the card in order to prolong its life. journald is quite > well-behaved in this regard since it offers the choice of a > temporary or permanent journal and limits the frequency of its > writes, except in emergencies. > > However, its permanent journal is written under /var/log/journal and > there is no way to configure the path as far as I am aware. I think > this is a problem. > > The reason is that on this type of machine people sometimes map > /var/log to RAM using tmpfs and then perhaps persist the logs > using a program like log2ram. When this is done, journald's > emergency writing capability is lost and crash analysis becomes > more difficult. > > Of course it is possible instead to redirect the log files of > programs individually to temporary memory using systemd-tmpfiles > wherever needed. But this involves reconfiguring each and every > program that uses /var/log both initially and whenever new programs > are installed. This is tedious not only in quantity but because > each program has a different detailed format of configuration file. > > So making /var/log into a tmpfs is a more attractive option. But > ideally the journal would be placed somewhere else in persistent > storage so its contents are available after a crash. This does not > seem to be possible through lack of a config option. > > Is my analysis correct? Are there any other ways to resolve this > difficulty? Otherwise, is it possible to consider a log location > config option for journald? > >>> > >>> Right now, when persistent mode is enabled journald will store its log > >>> data in /var/log. When it is disabled it will store things in /run/log > >>> instead. > >>> > >>> It has been requested that we add a hybrid mode that makes journald > >>> log to both locations at the same time, but filter by log priority so > >>> that log msgs higher than some priority go to one location and the > >>> ones below it go to the other. A patch like that would probably be > >>> relatively straight-forward and short. Would be happy to review/merge > >>> a patch for that. > >>> > >>> I think if that's implemented the log location should really stay > >>> unmodified: /var should be persistant and /run not, and there would be > >>> no need to remount any of those paths in a different way. > >> > >> Many thanks for the quick reply. I'm not clear how that resolves the > >> situation I explained, since it neither provides an alternative > >> persistent log location nor provides an alternative means of > >> arranging the logs of other programs. It doesn't seem to address my > >> issue at all? > > > > I don't understand the problem then? > > > > The journal only collects logs on stdout/stderr, syslog and its native > > protocol. Nothing else. It's not a tool that can collect arbitrary > > per-app logs that don't go via these transport hence. And it really > > shouldn't be. > > > > Also: /var is generally understood to be persistent, and /run to be > > volatile. If you want to redefine that you are welcome to, but of > > course you have to patch through all kinds of software then. > > As I understand it, you want /var/log to be tmpfs but /var/log/journal > to be persistent (as journald's write behaviour is considerate enough > for you not to worry about backing store fatigue etc)? > > Just as a less complex suggestion, you're presumably bootstrapping > /var/log to be on tmpfs at some point in early boot (i.e. before > journald is started or at least before it's being told to start writing > to /var/log/journal). If this is the case, then why don't you just amend > that bootstrapping to bind mount /var/log/journal to persistent storage > rather than letting it be just a subdir on the tmpfs? That way you can > get your mem-only syslog (or native logging) setup as you do now and get > persistent journal logs with only very minor changes (this could all be > done with appropriate systemd units and deps AFAICT). > > I can't remember off the top of my head what makes journald start > writing to /var/log/journal, so you may have to change that slightly > (e.g. it might notice is as soon as you mkdir it in the tmpfs but
Re: [systemd-devel] logging in RAM and journald configuration issue
This message was caught up in what looks like a freedesktop server crash so I'm reposting it slightly edited after other responses. Hopefully it will now reach the list and thread somewhat sensibly. > Lennart Poettering wrote on 20/03/2019 13:37: > > On Mi, 20.03.19 09:57, Dave Howorth (syst...@howorth.org.uk) wrote: > > > >>> On Di, 19.03.19 20:45, Dave Howorth (syst...@howorth.org.uk) > >>> wrote: > In the case of a machine that uses an SD card as its primary > backing store, it is desirable to reduce the number of write > operations to the card in order to prolong its life. journald is > quite well-behaved in this regard since it offers the choice of a > temporary or permanent journal and limits the frequency of its > writes, except in emergencies. > > However, its permanent journal is written under /var/log/journal > and there is no way to configure the path as far as I am aware. > I think this is a problem. > > The reason is that on this type of machine people sometimes map > /var/log to RAM using tmpfs and then perhaps persist the logs > using a program like log2ram. When this is done, journald's > emergency writing capability is lost and crash analysis becomes > more difficult. > > Of course it is possible instead to redirect the log files of > programs individually to temporary memory using systemd-tmpfiles > wherever needed. But this involves reconfiguring each and every > program that uses /var/log both initially and whenever new > programs are installed. This is tedious not only in quantity but > because each program has a different detailed format of > configuration file. > > So making /var/log into a tmpfs is a more attractive option. But > ideally the journal would be placed somewhere else in persistent > storage so its contents are available after a crash. This does > not seem to be possible through lack of a config option. > > Is my analysis correct? Are there any other ways to resolve this > difficulty? Otherwise, is it possible to consider a log location > config option for journald? > >>> > >>> Right now, when persistent mode is enabled journald will store > >>> its log data in /var/log. When it is disabled it will store > >>> things in /run/log instead. > >>> > >>> It has been requested that we add a hybrid mode that makes > >>> journald log to both locations at the same time, but filter by > >>> log priority so that log msgs higher than some priority go to one > >>> location and the ones below it go to the other. A patch like that > >>> would probably be relatively straight-forward and short. Would be > >>> happy to review/merge a patch for that. > >>> > >>> I think if that's implemented the log location should really stay > >>> unmodified: /var should be persistant and /run not, and there > >>> would be no need to remount any of those paths in a different > >>> way. > >> > >> Many thanks for the quick reply. I'm not clear how that resolves > >> the situation I explained, since it neither provides an alternative > >> persistent log location nor provides an alternative means of > >> arranging the logs of other programs. It doesn't seem to address my > >> issue at all? > > > > I don't understand the problem then? > > > > The journal only collects logs on stdout/stderr, syslog and its > > native protocol. Nothing else. It's not a tool that can collect > > arbitrary per-app logs that don't go via these transport hence. And > > it really shouldn't be. > > > > Also: /var is generally understood to be persistent, and /run to be > > volatile. If you want to redefine that you are welcome to, but of > > course you have to patch through all kinds of software then. > > As I understand it, you want /var/log to be tmpfs but /var/log/journal > to be persistent (as journald's write behaviour is considerate enough > for you not to worry about backing store fatigue etc)? Yes, a very good summary. > Just as a less complex suggestion, you're presumably bootstrapping > /var/log to be on tmpfs at some point in early boot (i.e. before > journald is started or at least before it's being told to start > writing to /var/log/journal). If this is the case, then why don't you > just amend that bootstrapping to bind mount /var/log/journal to > persistent storage rather than letting it be just a subdir on the > tmpfs? That way you can get your mem-only syslog (or native logging) > setup as you do now and get persistent journal logs with only very > minor changes (this could all be done with appropriate systemd units > and deps AFAICT). I'm not an expert, or even very familiar, with most of this stuff so I can't say for sure but yes I think it should be possible to do something like that. > I can't remember off the top of my head what makes journald start > writing to /var/log/journal, so you may have to change that slightly > (e.g.
Re: [systemd-devel] logging in RAM and journald configuration issue
On Mi, 20.03.19 18:08, Dave Howorth (syst...@howorth.org.uk) wrote: > > At present AFAICT the log2ram.service runs > Before=systemd-journald.service and various other services, so I think > that aspect is covered. I'd be careful with that. If you have a service that runs before journald you need to be very careful with the deps and also ensure you explicitly turn off stdout/stderr connection to journald, and avoid using the syslog() API (since that might deadlock, if you are ordered before journald, as this will trigger activation-by-socket but you are also blocking precisely that activation). I am not sure what this service does, but if all you want to do is make changes to /var/log/journal before journald writes there it should be sufficient to order it before systemd-journal-flush.service which is nowadays the clear boundary when journald starts writing to /var. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] logging in RAM and journald configuration issue
Lennart Poettering wrote on 20/03/2019 13:37: > On Mi, 20.03.19 09:57, Dave Howorth (syst...@howorth.org.uk) wrote: > >>> On Di, 19.03.19 20:45, Dave Howorth (syst...@howorth.org.uk) wrote: >>> In the case of a machine that uses an SD card as its primary backing store, it is desirable to reduce the number of write operations to the card in order to prolong its life. journald is quite well-behaved in this regard since it offers the choice of a temporary or permanent journal and limits the frequency of its writes, except in emergencies. However, its permanent journal is written under /var/log/journal and there is no way to configure the path as far as I am aware. I think this is a problem. The reason is that on this type of machine people sometimes map /var/log to RAM using tmpfs and then perhaps persist the logs using a program like log2ram. When this is done, journald's emergency writing capability is lost and crash analysis becomes more difficult. Of course it is possible instead to redirect the log files of programs individually to temporary memory using systemd-tmpfiles wherever needed. But this involves reconfiguring each and every program that uses /var/log both initially and whenever new programs are installed. This is tedious not only in quantity but because each program has a different detailed format of configuration file. So making /var/log into a tmpfs is a more attractive option. But ideally the journal would be placed somewhere else in persistent storage so its contents are available after a crash. This does not seem to be possible through lack of a config option. Is my analysis correct? Are there any other ways to resolve this difficulty? Otherwise, is it possible to consider a log location config option for journald? >>> >>> Right now, when persistent mode is enabled journald will store its log >>> data in /var/log. When it is disabled it will store things in /run/log >>> instead. >>> >>> It has been requested that we add a hybrid mode that makes journald >>> log to both locations at the same time, but filter by log priority so >>> that log msgs higher than some priority go to one location and the >>> ones below it go to the other. A patch like that would probably be >>> relatively straight-forward and short. Would be happy to review/merge >>> a patch for that. >>> >>> I think if that's implemented the log location should really stay >>> unmodified: /var should be persistant and /run not, and there would be >>> no need to remount any of those paths in a different way. >> >> Many thanks for the quick reply. I'm not clear how that resolves the >> situation I explained, since it neither provides an alternative >> persistent log location nor provides an alternative means of >> arranging the logs of other programs. It doesn't seem to address my >> issue at all? > > I don't understand the problem then? > > The journal only collects logs on stdout/stderr, syslog and its native > protocol. Nothing else. It's not a tool that can collect arbitrary > per-app logs that don't go via these transport hence. And it really > shouldn't be. > > Also: /var is generally understood to be persistent, and /run to be > volatile. If you want to redefine that you are welcome to, but of > course you have to patch through all kinds of software then. As I understand it, you want /var/log to be tmpfs but /var/log/journal to be persistent (as journald's write behaviour is considerate enough for you not to worry about backing store fatigue etc)? Just as a less complex suggestion, you're presumably bootstrapping /var/log to be on tmpfs at some point in early boot (i.e. before journald is started or at least before it's being told to start writing to /var/log/journal). If this is the case, then why don't you just amend that bootstrapping to bind mount /var/log/journal to persistent storage rather than letting it be just a subdir on the tmpfs? That way you can get your mem-only syslog (or native logging) setup as you do now and get persistent journal logs with only very minor changes (this could all be done with appropriate systemd units and deps AFAICT). I can't remember off the top of my head what makes journald start writing to /var/log/journal, so you may have to change that slightly (e.g. it might notice is as soon as you mkdir it in the tmpfs but before you do the bind mount), but I'm sure this would be simple enough. Just a thought. Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org
Re: [systemd-devel] logging in RAM and journald configuration issue
On Mi, 20.03.19 09:57, Dave Howorth (syst...@howorth.org.uk) wrote: > > On Di, 19.03.19 20:45, Dave Howorth (syst...@howorth.org.uk) wrote: > > > > > In the case of a machine that uses an SD card as its primary backing > > > store, it is desirable to reduce the number of write operations to > > > the card in order to prolong its life. journald is quite > > > well-behaved in this regard since it offers the choice of a > > > temporary or permanent journal and limits the frequency of its > > > writes, except in emergencies. > > > > > > However, its permanent journal is written under /var/log/journal and > > > there is no way to configure the path as far as I am aware. I think > > > this is a problem. > > > > > > The reason is that on this type of machine people sometimes map > > > /var/log to RAM using tmpfs and then perhaps persist the logs > > > using a program like log2ram. When this is done, journald's > > > emergency writing capability is lost and crash analysis becomes > > > more difficult. > > > > > > Of course it is possible instead to redirect the log files of > > > programs individually to temporary memory using systemd-tmpfiles > > > wherever needed. But this involves reconfiguring each and every > > > program that uses /var/log both initially and whenever new programs > > > are installed. This is tedious not only in quantity but because > > > each program has a different detailed format of configuration file. > > > > > > So making /var/log into a tmpfs is a more attractive option. But > > > ideally the journal would be placed somewhere else in persistent > > > storage so its contents are available after a crash. This does not > > > seem to be possible through lack of a config option. > > > > > > Is my analysis correct? Are there any other ways to resolve this > > > difficulty? Otherwise, is it possible to consider a log location > > > config option for journald? > > > > Right now, when persistent mode is enabled journald will store its log > > data in /var/log. When it is disabled it will store things in /run/log > > instead. > > > > It has been requested that we add a hybrid mode that makes journald > > log to both locations at the same time, but filter by log priority so > > that log msgs higher than some priority go to one location and the > > ones below it go to the other. A patch like that would probably be > > relatively straight-forward and short. Would be happy to review/merge > > a patch for that. > > > > I think if that's implemented the log location should really stay > > unmodified: /var should be persistant and /run not, and there would be > > no need to remount any of those paths in a different way. > > Many thanks for the quick reply. I'm not clear how that resolves the > situation I explained, since it neither provides an alternative > persistent log location nor provides an alternative means of > arranging the logs of other programs. It doesn't seem to address my > issue at all? I don't understand the problem then? The journal only collects logs on stdout/stderr, syslog and its native protocol. Nothing else. It's not a tool that can collect arbitrary per-app logs that don't go via these transport hence. And it really shouldn't be. Also: /var is generally understood to be persistent, and /run to be volatile. If you want to redefine that you are welcome to, but of course you have to patch through all kinds of software then. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] logging in RAM and journald configuration issue
> On Di, 19.03.19 20:45, Dave Howorth (syst...@howorth.org.uk) wrote: > > > In the case of a machine that uses an SD card as its primary backing > > store, it is desirable to reduce the number of write operations to > > the card in order to prolong its life. journald is quite > > well-behaved in this regard since it offers the choice of a > > temporary or permanent journal and limits the frequency of its > > writes, except in emergencies. > > > > However, its permanent journal is written under /var/log/journal and > > there is no way to configure the path as far as I am aware. I think > > this is a problem. > > > > The reason is that on this type of machine people sometimes map > > /var/log to RAM using tmpfs and then perhaps persist the logs > > using a program like log2ram. When this is done, journald's > > emergency writing capability is lost and crash analysis becomes > > more difficult. > > > > Of course it is possible instead to redirect the log files of > > programs individually to temporary memory using systemd-tmpfiles > > wherever needed. But this involves reconfiguring each and every > > program that uses /var/log both initially and whenever new programs > > are installed. This is tedious not only in quantity but because > > each program has a different detailed format of configuration file. > > > > So making /var/log into a tmpfs is a more attractive option. But > > ideally the journal would be placed somewhere else in persistent > > storage so its contents are available after a crash. This does not > > seem to be possible through lack of a config option. > > > > Is my analysis correct? Are there any other ways to resolve this > > difficulty? Otherwise, is it possible to consider a log location > > config option for journald? > > Right now, when persistent mode is enabled journald will store its log > data in /var/log. When it is disabled it will store things in /run/log > instead. > > It has been requested that we add a hybrid mode that makes journald > log to both locations at the same time, but filter by log priority so > that log msgs higher than some priority go to one location and the > ones below it go to the other. A patch like that would probably be > relatively straight-forward and short. Would be happy to review/merge > a patch for that. > > I think if that's implemented the log location should really stay > unmodified: /var should be persistant and /run not, and there would be > no need to remount any of those paths in a different way. Many thanks for the quick reply. I'm not clear how that resolves the situation I explained, since it neither provides an alternative persistent log location nor provides an alternative means of arranging the logs of other programs. It doesn't seem to address my issue at all? > Lennart > > -- > Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] logging in RAM and journald configuration issue
On Di, 19.03.19 20:45, Dave Howorth (syst...@howorth.org.uk) wrote: > In the case of a machine that uses an SD card as its primary backing > store, it is desirable to reduce the number of write operations to the > card in order to prolong its life. journald is quite well-behaved in > this regard since it offers the choice of a temporary or permanent > journal and limits the frequency of its writes, except in emergencies. > > However, its permanent journal is written under /var/log/journal and > there is no way to configure the path as far as I am aware. I think > this is a problem. > > The reason is that on this type of machine people sometimes map > /var/log to RAM using tmpfs and then perhaps persist the logs using a > program like log2ram. When this is done, journald's emergency writing > capability is lost and crash analysis becomes more difficult. > > Of course it is possible instead to redirect the log files of programs > individually to temporary memory using systemd-tmpfiles wherever > needed. But this involves reconfiguring each and every program that > uses /var/log both initially and whenever new programs are installed. > This is tedious not only in quantity but because each program has a > different detailed format of configuration file. > > So making /var/log into a tmpfs is a more attractive option. But > ideally the journal would be placed somewhere else in persistent > storage so its contents are available after a crash. This does not seem > to be possible through lack of a config option. > > Is my analysis correct? Are there any other ways to resolve this > difficulty? Otherwise, is it possible to consider a log location config > option for journald? Right now, when persistent mode is enabled journald will store its log data in /var/log. When it is disabled it will store things in /run/log instead. It has been requested that we add a hybrid mode that makes journald log to both locations at the same time, but filter by log priority so that log msgs higher than some priority go to one location and the ones below it go to the other. A patch like that would probably be relatively straight-forward and short. Would be happy to review/merge a patch for that. I think if that's implemented the log location should really stay unmodified: /var should be persistant and /run not, and there would be no need to remount any of those paths in a different way. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
