Re: [systemd-devel] Logging in an enterprise environment
On 12/10/2013 07:16 PM, Lennart Poettering wrote: As I understand it, journald is mend to log locally. Two methods to log centrally are, if I have understand it correctly: - mounting and merging through NFS - systemd-journal-gateway Whereby the first would be the preferred method. That would not be always acceptable for our clients. As I understood it there are environments where now 700+ systems are monitored from one log server. Using NFS for this would ask to much resources. I think that systemd-journal-gateway also would not be very handy for this because it is a pulling solution. To make things even more complicated: sometimes they want no local logging at all, all logging should be done remotely, so that not one user on the system could read the logs. Would something like this be possible to implement? If so, how much work would it be? Probably our company would be prepared to let programmers write this functionality. Journald's primary job is local logging, for complex logging setups we recommend installing a logging services like rsyslog. Would you not loose some of the advantages of journald this way? I understand that with using imjournal in rsyslog this probably could mostly be circumvented. I have seen that the local logging can be disabled with: Storage=none You can to centralized logging with the journal too, by simply making the journal files you want to look at accessible on the same machine. This could be done via NFS sharing, or by copying them to a central host via rsync or scp or even ftp, whatever you prefer. journalctl -m will then be able to coalesce them on display. NFS is not acceptable I am told, but I'll inquire about the other possibilities. By the way: is just copying possible? What if an event is logged at the moment the copy is done? Eventually journald should provide you with both a pull and a push model, however it currently onyl supports a pull model. Note that for Where can I information about that, or do you mean copying the files? large setups a pull model is actually the more deseriable since the centralized logging serve can then pull logs out of the cluster on its own terms and schedule. Such a setup in push scenario might make the log server vulnerable to flooding when some event happens across the whole cluster (example: the cluster networked is portscanned, or power state changes for all cluster machines at the same time). Sounds reasonable. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
On 12/11/2013 02:25 PM, Cecil Westerhof wrote: You can to centralized logging with the journal too, by simply making the journal files you want to look at accessible on the same machine. This could be done via NFS sharing, or by copying them to a central host via rsync or scp or even ftp, whatever you prefer. journalctl -m will then be able to coalesce them on display. NFS is not acceptable I am told, but I'll inquire about the other possibilities. By the way: is just copying possible? What if an event is logged at the moment the copy is done? I just had a little check. Copying is not acceptable. Logging on the log-server should be real-time. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
Am 11.12.2013 14:43, schrieb Cecil Westerhof: On 12/11/2013 02:25 PM, Cecil Westerhof wrote: You can to centralized logging with the journal too, by simply making the journal files you want to look at accessible on the same machine. This could be done via NFS sharing, or by copying them to a central host via rsync or scp or even ftp, whatever you prefer. journalctl -m will then be able to coalesce them on display. NFS is not acceptable I am told, but I'll inquire about the other possibilities. By the way: is just copying possible? What if an event is logged at the moment the copy is done? I just had a little check. Copying is not acceptable. Logging on the log-server should be real-time so just install rsyslog and you have the same as before rsyslog is running here on any machine and journald with Storage=none signature.asc Description: OpenPGP digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
On 12/11/2013 02:25 PM, Cecil Westerhof wrote: Eventually journald should provide you with both a pull and a push model, however it currently onyl supports a pull model. Note that for Where can I information about that, or do you mean copying the files? You probably mend systemd-journal-gateway? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
On 12/11/2013 02:52 PM, Reindl Harald wrote: I just had a little check. Copying is not acceptable. Logging on the log-server should be real-time so just install rsyslog and you have the same as before rsyslog is running here on any machine and journald with Storage=none I already suggested that. They want to talk more about it later. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
On Wed, 11.12.13 14:25, Cecil Westerhof (cecil.wester...@snow.nl) wrote: Journald's primary job is local logging, for complex logging setups we recommend installing a logging services like rsyslog. Would you not loose some of the advantages of journald this way? I understand that with using imjournal in rsyslog this probably could mostly be circumvented. Well, you cannot look at the logs with journalctl if you do this, but you cannot have it all.. By the way: is just copying possible? What if an event is logged at the moment the copy is done? That's fine. The journal parser is designed to handle half written files as good as it can. Eventually journald should provide you with both a pull and a push model, however it currently onyl supports a pull model. Note that for Where can I information about that, or do you mean copying the files? By pull model I was referring to the GET HTTP command to the journal gateway or copying via rsync/scp/ Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
On Wed, 11.12.13 14:43, Cecil Westerhof (cecil.wester...@snow.nl) wrote: On 12/11/2013 02:25 PM, Cecil Westerhof wrote: You can to centralized logging with the journal too, by simply making the journal files you want to look at accessible on the same machine. This could be done via NFS sharing, or by copying them to a central host via rsync or scp or even ftp, whatever you prefer. journalctl -m will then be able to coalesce them on display. NFS is not acceptable I am told, but I'll inquire about the other possibilities. By the way: is just copying possible? What if an event is logged at the moment the copy is done? I just had a little check. Copying is not acceptable. Logging on the log-server should be real-time. If you want logging on the server in real-time, then use rsyslog. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Logging in an enterprise environment
After giving a presentation about systemd/journald I am seen as the expert, so they come to me with the challenges they see. As I understand it, journald is mend to log locally. Two methods to log centrally are, if I have understand it correctly: - mounting and merging through NFS - systemd-journal-gateway Whereby the first would be the preferred method. That would not be always acceptable for our clients. As I understood it there are environments where now 700+ systems are monitored from one log server. Using NFS for this would ask to much resources. I think that systemd-journal-gateway also would not be very handy for this because it is a pulling solution. To make things even more complicated: sometimes they want no local logging at all, all logging should be done remotely, so that not one user on the system could read the logs. Would something like this be possible to implement? If so, how much work would it be? Probably our company would be prepared to let programmers write this functionality. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
Hi Cecil, - Original Message - After giving a presentation about systemd/journald I am seen as the expert, so they come to me with the challenges they see. As I understand it, journald is mend to log locally. Two methods to log centrally are, if I have understand it correctly: - mounting and merging through NFS - systemd-journal-gateway Whereby the first would be the preferred method. The first will fail shorts if the Environment is distributed somehow, mounting NFS drives is not always a option. We prefer some gateway solution which forwards the log messages to a aggregation host. That would not be always acceptable for our clients. As I understood it there are environments where now 700+ systems are monitored from one log server. Using NFS for this would ask to much resources. I think that systemd-journal-gateway also would not be very handy for this because it is a pulling solution. We think of forwarding the messages when arrived locally. But we should think about local caching as well. What about the aggregator or network is not available? To make things even more complicated: sometimes they want no local logging at all, all logging should be done remotely, so that not one user on the system could read the logs. Would something like this be possible to implement? If so, how much work would it be? Probably our company would be prepared to let programmers write this functionality. My open question would be: If you aggregate the logs, with your example from 700 hosts, should we use the Journald Fileformat on the central location as well? Does it scale? should we write one journal file per host? can jpurnalctl deal with 7000 files and the huge content? Further we would like to discuss the transport between the host and the aggregator. many ideas come along like AMQP, pure TCP, Syslog(seems to limited), HTTP? Holger systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Holger Winkelmann email: h...@travelping.com ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
On Tue, 10.12.13 12:16, Cecil Westerhof (cecil.wester...@snow.nl) wrote: After giving a presentation about systemd/journald I am seen as the expert, so they come to me with the challenges they see. As I understand it, journald is mend to log locally. Two methods to log centrally are, if I have understand it correctly: - mounting and merging through NFS - systemd-journal-gateway Whereby the first would be the preferred method. That would not be always acceptable for our clients. As I understood it there are environments where now 700+ systems are monitored from one log server. Using NFS for this would ask to much resources. I think that systemd-journal-gateway also would not be very handy for this because it is a pulling solution. To make things even more complicated: sometimes they want no local logging at all, all logging should be done remotely, so that not one user on the system could read the logs. Would something like this be possible to implement? If so, how much work would it be? Probably our company would be prepared to let programmers write this functionality. Journald's primary job is local logging, for complex logging setups we recommend installing a logging services like rsyslog. You can to centralized logging with the journal too, by simply making the journal files you want to look at accessible on the same machine. This could be done via NFS sharing, or by copying them to a central host via rsync or scp or even ftp, whatever you prefer. journalctl -m will then be able to coalesce them on display. Eventually journald should provide you with both a pull and a push model, however it currently onyl supports a pull model. Note that for large setups a pull model is actually the more deseriable since the centralized logging serve can then pull logs out of the cluster on its own terms and schedule. Such a setup in push scenario might make the log server vulnerable to flooding when some event happens across the whole cluster (example: the cluster networked is portscanned, or power state changes for all cluster machines at the same time). Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Logging in an enterprise environment
We currently use journal2gelf [1], which we also have a rewrite of that uses the native Python bindings to the journal. We're probably dumping our rewrite and adding journal integration to Beaver [2]. [1] https://github.com/systemd/journal2gelf [2] https://github.com/clifton/beaver ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
