Re: [systemd-devel] trust of kernel messages re-routed via journald
On Sun, Mar 4, 2012 at 11:37 PM, Lennart Poettering lenn...@poettering.net wrote: On Thu, 23.02.12 17:54, Rainer Gerhards (rgerha...@gmail.com) wrote: Hi, I am thinking on how to detect potential fake messages, claiming to be e.g. from the audit subsystem. Let's assume - auditd is stopped -- audit messages are put into the kernel log - journald controls /dev/kmsg and provides these via the the journal log socket to syslogd I presume you mean /proc/kmsg here, not /dev/kmsg? doh, of course ;) Note that on F17 (and most likely for much longer) systemd does not take control of /proc/kmsg and leaves that to syslog-ng/rsyslog. Sure, but the question was with a bit broader scope, assuming this will change in the future. - syslogd uses SCM_CREDENTIALS on the journald provided socket Question now: what pid will I see inside SCM_CREDENTIALS (0, 1, s/t else)? I assume I can use the pid to tell the difference between a real message and a faked one from some user process. Is that a correct assumption? You will see systemd's own PID if we have no other sensible PID to fill in. And if a message originates from the kernel we have no PID. OK, so it will be 1, I guess the same like systemd emitted messages. Does it sound decent enough to check if the PID is 1 AND the facility is kernel THEN this message is actually from the kernel log? I am asking because a couple of folks handle messages differently just because of their origin. So I think how to emulate this previous behavior when running under Journal. Thanks, Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] trust of kernel messages re-routed via journald
Hi, I am thinking on how to detect potential fake messages, claiming to be e.g. from the audit subsystem. Let's assume - auditd is stopped -- audit messages are put into the kernel log - journald controls /dev/kmsg and provides these via the the journal log socket to syslogd - syslogd uses SCM_CREDENTIALS on the journald provided socket Question now: what pid will I see inside SCM_CREDENTIALS (0, 1, s/t else)? I assume I can use the pid to tell the difference between a real message and a faked one from some user process. Is that a correct assumption? Thanks, Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [ANNOUNCE] systemd v38
On Wed, Jan 11, 2012 at 7:20 AM, Michael Biebl mbi...@gmail.com wrote: Am 11. Januar 2012 05:19 schrieb Lennart Poettering lenn...@poettering.net: Compatibility with classic syslog implementations is provided, via a socket /run/systemd/journal/syslog, to which all messages are forwarded, Most systems setup rsyslog to use /etc/rsyslog.d/, so dropping a small file like 00-systemd.conf there containing: $SystemLogSocketName /run/systemd/journal/syslog might do the trick. Haven't really tested and run the new journal yet, so there are probably more issues. I guess this is not even necessary if the syslogd gets its log socket via systemd (as rsyslog and syslog-ng do). Am I right here? Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [ANNOUNCE] systemd v38
On Wed, Jan 11, 2012 at 4:09 PM, Lennart Poettering lenn...@poettering.net wrote: rsyslog is actually too smart for us here, and will take the socket, but also open /dev/log in addition to that, which we want to avoid. I think rsyslog is actually doing the right thing here, though. It's the smart and the right thing to do to combine the list of sockets configured with the list of sockets passed. On second thought, you are right ;-). But it would probably good to have a mechanism that rsyslog (or anyone else) can query to find out the right path without config change. After all, /dev/log was not really configured but rather is what rsyslog uses as default. Obviously not a priority... Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Mon, Dec 5, 2011 at 1:40 AM, Kay Sievers kay.siev...@vrfy.org wrote: On Sat, Dec 3, 2011 at 20:41, Rainer Gerhards rgerha...@gmail.com wrote: I have digested our discussion now. Two questions came up: The first one is a bit direct and blunt, my apologies for that. But I want to make sure I put effort into the right place. Do you see any benefit in an interface like I described (reading and writing the journal from the syslogd)? Or do you think this is a needless effort? Oh sure, I think it can be very useful. We might need to find out how to identify such a stream of data in the journal, or if we should make it possible log things for a different machine-id. Your example like the SOHO router which pushes out syslog data could possibly get a machine-id assigned from the syslogd config that processes the data. We need to find out the details ... Syslog will not go away in many setups, the network log from other hosts which speak syslog, real boxes or the specialized hardware like a SOHO router, will speak the current syslog for a very long time in the future. And we don't want to speak real syslog with the journal; that's all stuff that should be done with syslog, and possibly if it can act as a bridge to the journal index, it sounds like a very nice option to have. This sounds reasonable to me as well. IMHO a bit problem in this process is how to keep trusted things trusted. But I understand that you currently have more important core things to do than tackle that beast or interop. I would be happy to provide feedback once you look at this issue. The second one is on /dev/log: is what is provided to the syslogd just like the real thing? I am specifically concerned if SCM_CREDENTIALS will properly identify the log emitter. We expect we are able to fake all the properties. If not we will need to make that possible, we didn't look into the details as of now, but the plan surely is to make /dev/log behave like it is without the journald interception. Yes, that would be very important, especially as the rate-limiting in rsyslog heavily depends on SCM_CREDENTIALS (as do some other things, but rate-limiting is the most important one IMHO). Thanks again, Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Sat, Dec 3, 2011 at 12:54 PM, Kay Sievers kay.siev...@vrfy.org wrote: Well, that's not making it hard to access information, it's trying hard not to break established interfaces. If syslog *could* be extended in a reasonable way, we wouldn't need a journal interface at all. Adding any information might break tools, so we can only offer the extended stuff in a new interface. I think this boils down to the fundamental difference in our view on syslog, so I guess it does not make much sense to argue here in this context. But Ifind it interesting to hear that you postulate that syslog logging is absolutely static and can not be changed at all. If one follows that claim, it is natural to see there can be no evolution, as evolution requires change by definition. Anyhow, as you say, there are interfaces that the syslog world can get hold of the complete journal data, and that is what I was originally interested to learn. If you already supply the event you gather via the log socket, why not provide the complete set of information (probably as an opt-in feature)? That way the syslogd would not need a specific input capability (input module in rsyslog terms) to capture the log data. Things could just remain as is. You mean the native log messages only or all messages, including the ones journald retrieves from /dev/log I understood from your posting, that you provide everything that is logged, not matter what API is used, but only the non-structured data part. We just forward the exact same information syslog information we have intercepted, and we will also the syslog compatible part of the native journal information we received. But we are absolutely not allowed to change the format of the messages we from /dev/log. Note that I was following your initial answer that the syslogd does not need to talk to the journal API because it receives all it needs via the system log socket. Now this seems not to be true, as important parts are missing. As such my initial assumption that the journal API must be used was true. I wanted to clarify that point. Also, I do not understand, even if syslog would be totally static, why it could break anything if you provide the full information, including structured data, for a message you received via the *new* API. Obviously, as it came in via the new API, the message did not exist before. It is hard to envision that you can break backwards compatibility for something that formerly did not even exist. Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Sat, Dec 3, 2011 at 5:18 PM, Kay Sievers kay.siev...@vrfy.org wrote: You can just add a tiny bridge or make rsyslog receive the proper data which has no fixed wire format at all, but is an API we can extend as needed. Honestly, I really don't understand your claims, all will be available to you with just a few lines of code. Syslog should start integrating with the system... Frankly, that's why I am asking in the first place. But it is unavoidable to clarify some things while doing so. One more question on the integration part. We now know that rsyslog needs to read the journal and convert that into proper syslog format. Given my experience with the Windows Event Log, I know that I need some handle to tell the system please give me everything from checkpoint x and above. In Windows, I have unique keys for such checkpoints, so that I know exactly to query from where. Is it your intention to provide such a handle as well (I know I could check the code, but that would give me just the current state and not what your intention towards that thing is). It is important that the handle is consistent and valid across reboot sessions. I guess you could use the hash? Maybe that's the way things will work? Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Sat, Dec 3, 2011 at 6:12 PM, Kay Sievers kay.siev...@vrfy.org wrote: It's called a 'cursor', a text string that is portable even across machines, it will point to the closest possible record in the stream. In the idea it's like the git commit hash that identifies a commit, in the journal context it's just not a number but a magic text string with multiple values which are matched against the data in the journal streams. Excellent, thanks. That was what I was looking for. Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
I have digested our discussion now. Two questions came up: The first one is a bit direct and blunt, my apologies for that. But I want to make sure I put effort into the right place. Do you see any benefit in an interface like I described (reading and writing the journal from the syslogd)? Or do you think this is a needless effort? The second one is on /dev/log: is what is provided to the syslogd just like the real thing? I am specifically concerned if SCM_CREDENTIALS will properly identify the log emitter. Thanks, Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Linux Journal API/client lib
Hi there, as you probably know, I am not a big fan of the journald proposal, but that's not the point of my question. I am thinking about how to integrate journal data into a syslog logging solution. So I have a couple of questions in regard to accessing the journal. The most important one is how to read information from it. From what I have read, a client API is planned. I know git has working code, but this code is tied into systemd and is not a lib. So I wonder how a non-systemd app would gain access to journal data. Do you actually plan a client library (-ljournal) ? Or do you plan to make this available as a core OS service (and thus via a system call interface)? Or anything else? Also, in order to read the complete journal, is it necessary to run under the root user or can lower privileges be used (and, if so, which)? These questions were mostly related to the syslogd pulling data from the journal to transmit it some remote site (or do some other transformation). Now let's assume the contrary, syslogd receives data via the network and wants to store it inside the journal. Will there be an API provided for that case as well? Looking forward to any insight you can provide. Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Fri, Dec 2, 2011 at 2:49 PM, Kay Sievers kay.siev...@vrfy.org wrote: On Fri, Dec 2, 2011 at 13:59, Rainer Gerhards rgerha...@gmail.com wrote: as you probably know, I am not a big fan of the journald proposal, but that's not the point of my question. I am thinking about how to integrate journal data into a syslog logging solution. You know that the syslog daemon will still see exactly the same log messages from all clients as it did before, right? The /dev/log file descriptor that systemd passed to the syslog daemon at startup, will still carry all the same things regardless of journald's actions. Does that mean /dev/log will also receive messages submitted via the *new* API you define? If so, is the format documented somewhere (or intended to be)? So I have a couple of questions in regard to accessing the journal. The most important one is how to read information from it. From what I have read, a client API is planned. I know git has working code, but this code is tied into systemd and is not a lib. So I wonder how a non-systemd app would gain access to journal data. Do you actually plan a client library (-ljournal) ? Yes, a shared library will be provided. It's probably a bit early, but: do you have some idea when this will be released (maybe around F17)? Or do you plan to make this available as a core OS service (and thus via a system call interface)? What do you mean? The journal is just files on disk (or tmpfs), and the lib can map the file and provide indexed access to it. I wondered if you intend to move the journal core into the kernel itself. I hadn't expected that, and it's not the case, as you just said. Still wanted to make sure I understood right. Or anything else? Also, in order to read the complete journal, is it necessary to run under the root user or can lower privileges be used (and, if so, which)? All entries are written to individual journals, a separate one for the system and one for every login uid (user) that submitted entries. Every login uid can read their own journal, but not directly write it. But there will be one journal that a root admin can pull for all log entries? Or does this mean that, in order to obtain all entries, the system journal file plus all journal files for all users must be read? If so, is it intended that the API/lib handles that? All services log into the system journal. The files of the system journal are owned by a group, you need to be member of that group to access them. These questions were mostly related to the syslogd pulling data from the journal to transmit it some remote site (or do some other transformation). Now let's assume the contrary, syslogd receives data via the network and wants to store it inside the journal. Will there be an API provided for that case as well? Yes, sure, it can just forward things to the journal. Along with what they log, they will just have some metadata of the forwarder added. So now let's assume I have pulled some log messages from system A, and transport these via syslog to system B. Now I want to consolidate the log on system B. So what I need is to have an exact duplicate of what is present on A also present on B (especially the metadata). That means I can write into B's journal exactly what was on A, including the *trusted fields*? (or let's for a moment assume that A does not run journald, but I know A's hostname via RFC5425 X.509-based auth and so this info is known to be correct - many scenarios along these lines). ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Fri, Dec 2, 2011 at 4:39 PM, Kay Sievers kay.siev...@vrfy.org wrote: On Fri, Dec 2, 2011 at 16:14, Rainer Gerhards rgerha...@gmail.com wrote: On Fri, Dec 2, 2011 at 2:49 PM, Kay Sievers kay.siev...@vrfy.org wrote: On Fri, Dec 2, 2011 at 13:59, Rainer Gerhards rgerha...@gmail.com wrote: as you probably know, I am not a big fan of the journald proposal, but that's not the point of my question. I am thinking about how to integrate journal data into a syslog logging solution. You know that the syslog daemon will still see exactly the same log messages from all clients as it did before, right? The /dev/log file descriptor that systemd passed to the syslog daemon at startup, will still carry all the same things regardless of journald's actions. Does that mean /dev/log will also receive messages submitted via the *new* API you define? If so, is the format documented somewhere (or intended to be)? /dev/log will be read by journald. The syslog.socket filedescriptor that the syslog daemon receives, will be provided by journald and have all the messages which are received by /dev/log and the human readable part that is received over the native journal interface. This excludes the metadata, right? I'd say you should reconsider this, probably as a config option (metadata is especially useful if developers use the new capabilities and do not provide any human-readable form, or at least with partial info only in the human readable form). I guess if this is provided, there would hardly be a need for syslogd to pull the information from the journal. But there will be one journal that a root admin can pull for all log entries? Or does this mean that, in order to obtain all entries, the system journal file plus all journal files for all users must be read? If so, is it intended that the API/lib handles that? The library already merges all files the caller has access (unix file permissions) to. It's transparent to the reader, if it can read them, they will be included in the stream. That's nice and useful. Yes, sure, it can just forward things to the journal. Along with what they log, they will just have some metadata of the forwarder added. So now let's assume I have pulled some log messages from system A, and transport these via syslog to system B. Now I want to consolidate the log on system B. So what I need is to have an exact duplicate of what is present on A also present on B (especially the metadata). That means I can write into B's journal exactly what was on A, including the *trusted fields*? (or let's for a moment assume that A does not run journald, but I know A's hostname via RFC5425 X.509-based auth and so this info is known to be correct - many scenarios along these lines). We haven't thought about any of the details how to handle possible trusted log merge. It's surely possible to do something like that, but we have no specific ideas as of now. In the end it's again a bit like git, and the model journal can do across the network is like 'git pull' from other hosts. So there is certainly the possibility of having a 'syslog journal gateway' that provides the syslog 'commits' which are to be merged. IMHO if the journal intend to be *the* log store on Linux, it needs to have such import facility. Even on a low-end system, you will probably want to have logs from your soho router included. Then it make much more sense to query that database. The problem is that with this the trustedness of trusted fields must somehow be ensured. And this is where it get's complicated. Maybe it's not something to target for the initial release, but definitely something that you should think about. I admit that I am quite interested in that part of the journal idea -- it will probably save me from writing a full-blow, fast-query-time log store. I am not sure if that's on your agenda at all (the current circular buffer is questionable for some of these use cases). But such a central store, with a standardized API, would probably make sense to support in log analysis projects (I am specifically thinking about our log analyzer, a web interface, but there are probably many more). In that sense, I'd (ab?)use journal as a database,even for cases where I would not be interested at all in local events... (but of course there are many factors to consider, like volume of data, etc, etc. - just wanted to convey the idea). Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Fri, Dec 2, 2011 at 6:59 PM, Kay Sievers kay.siev...@vrfy.org wrote: On Fri, Dec 2, 2011 at 17:02, Rainer Gerhards rgerha...@gmail.com wrote: On Fri, Dec 2, 2011 at 4:39 PM, Kay Sievers kay.siev...@vrfy.org wrote: /dev/log will be read by journald. The syslog.socket filedescriptor that the syslog daemon receives, will be provided by journald and have all the messages which are received by /dev/log and the human readable part that is received over the native journal interface. This excludes the metadata, right? I'd say you should reconsider this, probably as a config option (metadata is especially useful if developers use the new capabilities and do not provide any human-readable form, or at least with partial info only in the human readable form). I guess if this is provided, there would hardly be a need for syslogd to pull the information from the journal. Well, if syslogd, or any other consumer, is interested in the metadata, it should not rely in /dev/log. /dev/log will probably stay what it is which is mostly plain old syslog with a header and a timestamp and the human readable string. All stuff that wants the metadata should use the proper API and get the records from there. The '/dev/log proxy' is just to ensure full syslogd compatibility, not to provide any new data which do not really fit into the plaintext file format. Why are you making it deliberately hard to interoperate? If you already supply the event you gather via the log socket, why not provide the complete set of information (probably as an opt-in feature)? That way the syslogd would not need a specific input capability (input module in rsyslog terms) to capture the log data. Things could just remain as is. What you call the plaintext file format issue is being solved anyhow in the meantime. Look at RFC5424 or CEE. So how to encode the fields is actually not a question at all. If you provide proper formatting, the syslogd would not even need to be aware of what's going on... (at least for CEE, rsyslog also has a parser for structured data in legacy syslog, but that's non-standard). Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Linux Journal API/client lib
On Fri, Dec 2, 2011 at 7:47 PM, Kay Sievers kay.siev...@vrfy.org wrote: Well, if syslogd, or any other consumer, is interested in the metadata, it should not rely in /dev/log. /dev/log will probably stay what it is which is mostly plain old syslog with a header and a timestamp and the human readable string. All stuff that wants the metadata should use the proper API and get the records from there. The '/dev/log proxy' is just to ensure full syslogd compatibility, not to provide any new data which do not really fit into the plaintext file format. Why are you making it deliberately hard to interoperate? Making what hard? Interoperate? What do you mean? Syslog can very easily be a consumer of the journal data, if it's interested in it. I mean that you make it harder for the syslogd to read the journal than necessary. Granted, I started this discussion with the assumption that I needed to write some input plugin that reads the journal and passes that on (just like we do with the Windows Event Log). But then I was deligthed to hear that you provide kind of a bridge, where the system log socket (as seen by syslogd) receives data both from syslog() as well as your new API. HOWEVER, that log socket seems to be a a second-class citizen, in that it does not receive all the information, only the human-readable text. So the most interesting part (the structured data) is not passed. So in order to obtain the full information, I still need to read the journal via an additional plugin, what strictly would not be necessary. This is why I conclude you are making it hard(er than necessary) to interoperate. I suggest that you review this decision. But, of course, it's yours. If you already supply the event you gather via the log socket, why not provide the complete set of information (probably as an opt-in feature)? That way the syslogd would not need a specific input capability (input module in rsyslog terms) to capture the log data. Things could just remain as is. You mean the native log messages only or all messages, including the ones journald retrieves from /dev/log I understood from your posting, that you provide everything that is logged, not matter what API is used, but only the non-structured data part. What you call the plaintext file format issue is being solved anyhow in the meantime. Look at RFC5424 or CEE. So how to encode the fields is actually not a question at all. If you provide proper formatting, the syslogd would not even need to be aware of what's going on... (at least for CEE, rsyslog also has a parser for structured data in legacy syslog, but that's non-standard). The journal is not meant to replace syslog, if syslog is needed, nor to act as a proxy to enrich /dev/log messages. We need to change as little as possible, and that includes keeping the same format on the output (the fd handed to syslog) as we get at the input side (/dev/log). That sound much like we got a better interface, but we don't like to share it ;) If one day glibc changes or there is another generally useful new syslog client API, we can do alll that, but as of now, I'm convinced that the journal should just make it easy to get all the information out, but it should not try to add stuff to established interfaces, and it should not natively speak any syslog enhanced format. That stays as the job for syslog. The key point is that you *do not need a new syslog client API*, as for example CEE was specially modelled to live within the legacy syslog space. For RFC5424 it would be useful to have a new API, but we (rsyslog) already got a parser to support RFC5424 STRUCTURED-DATA within legacy messages. I agree, though, that is it somewhat clumpsy to do this on top of the syslog() API. But this is exactly where I thought your effort intended to help. If you known the name-value pairs, why not format the syslog message in one of these standard formats? Don't get me wrong: I agree that some syslog implementation may have problems with the somewhat longer messages. However, on Linux we have syslog-ng and rsyslog in most cases. Both handle this very well (syslog-ng just recently added a JSON parser to support CEE). I think it would be useful to pass a standards-format syslog message with all the data you have, provided the system *is configured* to do so. You may turn this off by default (a conservative choice), but I don't see a reason to *prevent* passing this information that is readly available. Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] what does systemd do if the syslogd errs on start?
On Wed, Jul 6, 2011 at 4:17 PM, Lennart Poettering lenn...@poettering.net wrote: On Wed, 06.07.11 16:11, Rainer Gerhards (rgerha...@gmail.com) wrote: Yes, if rsyslog dies, then systemd will notice it. As soon as there is traffic on the /dev/log socket (which might be right-away) it will then start the syslog-kmsg bridge again. Will systemd make any attempt to restart syslogd if it failed? I think there is pros and cons in it: a) if syslogd failed due to config error, a restart is bad b) if syslogd failed for stability reason, a reasonable number of restarts may make sense I assume that the user can always make systemd restart syslogd, e.g. after fixing a conf problem. So we should be reasonably safe here that nothing is completely lost when rsyslog dies. From a client perspective nothing of this is visible. Clients can jus use syslog(), and this will go to rsyslog as long as it is started, and to kmsg before rsyslog started and after it died (regardless whether that was abnormally or cleanly). Let's make sure we are on the same line. Sequece of events: 1. systemd starts syslogd 2. syslogd parses config, detects errors 3. syslogd logs config errors via syslog() 4. syslogd terminates Will that work? Note that at #3, syslogd has not yet terminated, so from a systemd POV it will probably look healthy. Having said that, would it make sense to somehow make syslogd flag systemd that it had a successful startup and is now ready to process messages? Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] what does systemd do if the syslogd errs on start?
Hi all, I hope this is the right list. I wonder what systemd does if the syslogd does not start when told to do so. Reason behind this question: in rsyslog, I try hard to record messages even if rsyslog.conf is screwed up. For that reason, I accept partially complete configs. And if things go really bad and I can not get anything that looks working, I startup a special, hardcoded, minimal config. All of this just in an effort to prevent log message loss. Now with systemd around, I hope I can do the cleaner thing and just err out and terminate rsyslogd. That would probably alert users much better. It would also clean up the code and be less surprising to users (it either works fully correct or not at all - what you usually expect). I assume that in this situation systemd takes the log socket over again. Am I right with that? What would be the best way to log a message to systemd in such a situation? Via the usual syslog() mechanism (with rsyslog being a client in this case)? Thanks, Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] rsyslog drops messages in debug mode
-Original Message- From: Lennart Poettering [mailto:lenn...@poettering.net] Sent: Wednesday, April 20, 2011 2:33 AM To: Michael Biebl Cc: Mike Kazantsev; systemd-devel@lists.freedesktop.org; Rainer Gerhards Subject: Re: [systemd-devel] rsyslog drops messages in debug mode On Fri, 18.03.11 07:42, Michael Biebl (mbi...@gmail.com) wrote: 2011/3/18 Mike Kazantsev mk.frag...@gmail.com: On Fri, 18 Mar 2011 07:03:01 +0100 Michael Biebl mbi...@gmail.com wrote: Hi, when booting with systemd.log_level=debug, I get Mar 18 06:39:26 pluto rsyslogd-2177: imuxsock begins to drop messages from pid 1 due to rate-limiting ... Mar 18 06:39:31 pluto rsyslogd-2177: imuxsock lost 127 messages from pid 1 due to rate-limiting Note: I didn't explicitly use systemd.log_target=kmsg I believe it's an rsyslog (imuxsock module) newer feature to limit flow from extra-verbose apps by default, which you can tune or completely bypass, as described in the docs: I know that (and I forgot to add that I use rsyslog 5.7.8). My point rather is, that for a default rsyslog + systemd installation imho this should not happen. I'm wondering if the default values in rsyslog are too aggressive in that regard? That's why I suggest using systemd.log_level=debug in combination with systemd.log_target=kmsg. But yeah, maybe the default should be raised a little in rsyslog. Or maybe rsyslog should allow more debug info than other info by defualt? Dunno. Maybe file a bug against rsyslog? I am not opposed to raising default limits (remember that you can always tune them via the config). But what's the right level? After all they are there as a guard against too noisy sources. If we raise the level too high, that won't work. I think there really is a conflict in these cases... Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-logger and external syslog daemon
-Original Message- From: Lennart Poettering [mailto:lenn...@poettering.net] Sent: Thursday, March 17, 2011 10:38 PM To: Rainer Gerhards Cc: Michael Biebl; Andrey Borzenkov; systemd- de...@lists.freedesktop.org Subject: Re: [systemd-devel] systemd-logger and external syslog daemon On Thu, 17.03.11 08:38, Rainer Gerhards (rgerha...@hq.adiscon.com) wrote: You mean a new udev/dracut/systemd on an old kernel? The messages they print would look a bit weird if they are used together with log msg timestamping the way the kernel does it, since the kernel doesn't recognize the prefix. (See Kay's post about this). But besides these cosmetic issues nothing should really go wrong. (I wonder if we can find a nice way to detect whether the kernel is new enough for this, so that we could strip the facility automatically for older ones. Explcitily checking for kernel versions at runtime is evil though... I can't think of a good way though...) Wouldn't it work to check if there is a PRI right at the start of the message? I think that it is actual user data would be extremely improbable, so this should be a good enough indication. That way, we could pull the PRI even without the kernel patch (but, granted, it is kind of an interface change...). Hmm? The question is how we can detect whether it is safe to write messages to kmsg with PRI values with more than 3 bits. 2.6.39 and above will be able to handle that properly, even if you enable per-line printk kernel timestamping. On 2.6.38 only 3-bit-PRI values will look good if you use printk kernel timestamping. Probably I misunderstood the answer to what happens on a kernel without that patch if a full PRI is written?. I understood the answer was the PRI is moved into the message. So 123msg would actually become 1 [TS] 123Msg From your answer I deduce this understanding is incorrect. So what will then happen on kernels without that patch if printk is provided a message 123MSg? Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-logger and external syslog daemon
-Original Message- From: Lennart Poettering [mailto:lenn...@poettering.net] Sent: Thursday, March 17, 2011 12:31 AM To: Michael Biebl Cc: Andrey Borzenkov; systemd-devel@lists.freedesktop.org; Rainer Gerhards Subject: Re: [systemd-devel] systemd-logger and external syslog daemon On Wed, 16.03.11 07:18, Michael Biebl (mbi...@gmail.com) wrote: 2011/3/16 Lennart Poettering lenn...@poettering.net: On Sat, 12.03.11 16:31, Andrey Borzenkov (arvidj...@mail.ru) wrote: Attached patch preserves full syslog facility marker and simply emits it back. So userspace is free to feed any facility it deems appropriate, not only LOG_USER. This is a good approach. Kay has independently prepped a patch for this now and it is already on its way into the kernel. It is hence very likely that pretty soon there's no reason anymore to strip the facility from the log messages before echoing them into /proc/kmsg. As soon as that patch is in the standard kernel I'll fix systemd to no longer strip the facility. Kay will do the same for udev. And Harald hopefully for Dracut too. And then all messages should contain the same amount of information regardless which way the took to the syslog daemon: directly via the /dev/log socket, or indirectly via the kmsg queue. What happens if you run udev/dracut/systemd on a kernel without this patch? You mean a new udev/dracut/systemd on an old kernel? The messages they print would look a bit weird if they are used together with log msg timestamping the way the kernel does it, since the kernel doesn't recognize the prefix. (See Kay's post about this). But besides these cosmetic issues nothing should really go wrong. (I wonder if we can find a nice way to detect whether the kernel is new enough for this, so that we could strip the facility automatically for older ones. Explcitily checking for kernel versions at runtime is evil though... I can't think of a good way though...) Wouldn't it work to check if there is a PRI right at the start of the message? I think that it is actual user data would be extremely improbable, so this should be a good enough indication. That way, we could pull the PRI even without the kernel patch (but, granted, it is kind of an interface change...). Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-logger and external syslog daemon
-Original Message- From: Andrey Borzenkov [mailto:arvidj...@mail.ru] Sent: Friday, March 11, 2011 8:38 AM To: Michael Biebl Cc: Mike Kazantsev; systemd-devel@lists.freedesktop.org; Rainer Gerhards Subject: Re: [systemd-devel] systemd-logger and external syslog daemon On Fri, Mar 11, 2011 at 10:03 AM, Michael Biebl mbi...@gmail.com wrote: For me the log messages actually look slightly different, as I also get the kernel timestamp and I also noticed a different problem: Mar 11 07:56:27 pluto kernel: imklog 5.7.8, log source = /proc/kmsg started. Mar 11 07:56:27 pluto rsyslogd: [origin software=rsyslogd swVersion=5.7.8 x-pid=25093 x-info=http://www.rsyslog.com;] start Mar 11 07:56:27 pluto kernel: [ 5913.491848] michael[24089]: foo Mar 11 07:56:27 pluto kernel: [ 5918.029738] michael[24911]: bar Mar 11 07:56:27 pluto kernel: [ 5921.140864] michael[25078]: baz As you can see, when rsyslog starts up and flushes the kmsg queue, the log messages all have the same timestamp (Mar 11 07:56:27) and they come after the rsyslog startup message, although they were logged before the rsyslog start. But that was the case for as long as I remember. It is not systemd specific in any way. Lennart argues, that this should be handles within the syslogd (in this case rsyslog 5.7.8), which should use the kernel time stamp to compute the correct time when the log message occurred. Sounds quite reasonable :) What would be also really nice - some systemd specific marker so rsyslog could extract syslogd messages from kmsg. Not sure if it is really doable without some gross kernel hack though. Special severity level may be ... PRINTK_SYSTEMD? :) There is also a subtle issue with the current systemd implementation, and that could potentially solved by such a setting. Systemd shuffles the system log socket to the kernel log. That is nice, because we have logging available right from the system start. However, in rsyslog users can configure different rules based on the log source. The issue now is that what used to be the local log socket source now becomes the kernel log source. I don't think this causes many problems in almost all environments, and I guess it would require some non-trivial magic in rsyslog to handle the situation (and I am not sure it is worth that). But I wanted to mention this point ;) Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-logger and external syslog daemon
-Original Message- From: Lennart Poettering [mailto:lenn...@poettering.net] Sent: Friday, March 11, 2011 5:06 PM To: Rainer Gerhards Cc: Michael Biebl; Andrey Borzenkov; Mike Kazantsev; systemd- de...@lists.freedesktop.org Subject: Re: [systemd-devel] systemd-logger and external syslog daemon On Fri, 11.03.11 08:09, Rainer Gerhards (rgerha...@hq.adiscon.com) wrote: As you can see, when rsyslog starts up and flushes the kmsg queue, the log messages all have the same timestamp (Mar 11 07:56:27) and they come after the rsyslog startup message, although they were logged before the rsyslog start. Lennart argues, that this should be handles within the syslogd (in this case rsyslog 5.7.8), which should use the kernel time stamp to compute the correct time when the log message occured. Rainer, can you share any insight on this matter? Lennart recommended that to me and I had some code in place to do it. However, at that time this did not work because the kernel did not record that timestamp. This was added a while later, but I did not yet revisit that issue. I was a bit hesitant to dig into this issue as I found no simple enough method to setup a system with systemd (I know it's important, but there are many other important things as well...). I'll see that I can at least see what kernel patch needs to be present. Nah, these are actually two different things. The SO_TIMESTAMP stuff does not matter in this context. What I'd like to see that SO_TIMESTAMP is used when messages come in via /dev/log. And for messages coming in from /proc/kmsg it would be cool to parse the kernel timestamps that (optionally) are in the message prefix in the [] part. Got it -- but optionally does not sound too good. What if systemd's minimal syslog implementation would guarantee that a timestamp is written for forwarded logs? Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-logger and external syslog daemon
-Original Message- From: Lennart Poettering [mailto:lenn...@poettering.net] Sent: Friday, March 11, 2011 5:17 PM To: Rainer Gerhards Cc: Andrey Borzenkov; Michael Biebl; Mike Kazantsev; systemd- de...@lists.freedesktop.org Subject: Re: [systemd-devel] systemd-logger and external syslog daemon On Fri, 11.03.11 09:55, Rainer Gerhards (rgerha...@hq.adiscon.com) wrote: Sounds quite reasonable :) What would be also really nice - some systemd specific marker so rsyslog could extract syslogd messages from kmsg. Not sure if it is really doable without some gross kernel hack though. Special severity level may be ... PRINTK_SYSTEMD? :) There is also a subtle issue with the current systemd implementation, and that could potentially solved by such a setting. Systemd shuffles the system log socket to the kernel log. That is nice, because we have logging available right from the system start. However, in rsyslog users can configure different rules based on the log source. The issue now is that what used to be the local log socket source now becomes the kernel log source. I don't think this causes many problems in almost all environments, and I guess it would require some non-trivial magic in rsyslog to handle the situation (and I am not sure it is worth that). But I wanted to mention this point ;) I think rules like this should look for the facility field, and we should allow facility bits in the kmsg messages, so that userspace messages can clearly mark themselves as such. There are too few facilities for this to work. Also, the engine can bind to different rulesets depending on the message origin. It is not trivial to re-route messages in this context. Granted, that's not a problem for a typical system, but it can be one in high-end environments. For rsyslog, it would mean I need to find some kind of an inter-module interface, where imklog shuffles some of the messages to imuxsock. Maybe it would make more sense if imklog in such cases simply forwards the messages to the system log socket (which in that scenario should be bound to rsyslog). But I don't think this is a really pressing problem -- we better see at least one instance of it in reality before trying to fix it... Rainer ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-logger and external syslog daemon
-Original Message- From: Lennart Poettering [mailto:lenn...@poettering.net] Sent: Friday, March 11, 2011 5:22 PM To: Rainer Gerhards Cc: Michael Biebl; Andrey Borzenkov; Mike Kazantsev; systemd- de...@lists.freedesktop.org Subject: Re: [systemd-devel] systemd-logger and external syslog daemon On Fri, 11.03.11 17:08, Rainer Gerhards (rgerha...@hq.adiscon.com) wrote: Lennart recommended that to me and I had some code in place to do it. However, at that time this did not work because the kernel did not record that timestamp. This was added a while later, but I did not yet revisit that issue. I was a bit hesitant to dig into this issue as I found no simple enough method to setup a system with systemd (I know it's important, but there are many other important things as well...). I'll see that I can at least see what kernel patch needs to be present. Nah, these are actually two different things. The SO_TIMESTAMP stuff does not matter in this context. What I'd like to see that SO_TIMESTAMP is used when messages come in via /dev/log. And for messages coming in from /proc/kmsg it would be cool to parse the kernel timestamps that (optionally) are in the message prefix in the [] part. Got it -- but optionally does not sound too good. What if systemd's minimal syslog implementation would guarantee that a timestamp is written for forwarded logs? Well, if the kernel adds timestamps anyway, then there's little point to add another set of timestamps from userspace, snce then we might end up with two timestamps in each message. And it is difficult to figure out whether kernel-side timestamping is on. Well, I could check a special cookie for the systemd timestamp. I am *not* concerned so much about regular kernel messages. What concerns me are messages that are originally destined to the system log socket. Also note that it may be dangerous to change the output format of syslogd. Various processes seem to die in that case (at least that's why almost all distros still have the silly old-style timestamp set as default, because a variety of tools abort if RFC3339 timestamps are being used...). All big distros enable printk timestamping by default. That means this is an opt-out and not an opt-in feature nowadays. That probably means we shouldn't try to ignore the user configuration and add in timestamps when the kernel doesn't genreate them anyway. Also, note that systemd is not the only one logging to /dev/kmsg. The kernel timestamping covers all those sources equally. As I said, I am concerned about the forwarded messages, for which I think systemd ist he only source. Rainer Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
