Re: Journald's qualities (was: Selective rotation of journald logs)
Dnia 2024-02-23, o godz. 15:05:52 Nicholas Geovanis napisał(a): > On Fri, Feb 23, 2024, 2:57 PM Dan Ritter wrote: > > > Stefan Monnier wrote: > > > Makes one wonder why they don't use naive append-only "plain > > > text" logs (tho with appropriate delimiters (maybe some kind of > > > CSV) to make searches more reliable than with old-style plain > > > text logs)? > > > > > > What are the advantages of journald's representation? > > > I mean, to justify the slow search and large disk space usage, > > > there is presumably some upside for some use cases. I can see > > > some weak argument against Sqlite based on the size of Sqlite, > > > but what are the advantages of journald's representation compared > > > to a naive one? > > > > > > systemd's design philosophy, observed from the outside, goes > > like this: > > > > bunch trimmed. > > Exactly correct in my view. Systemd's use-case is the desktop, not the > server in the datacenter. They will be using log-aggregation software > in the datacenter anyway so no use for systemd logging. We don't > install desktop software on servers either, no X Windows, no gnome, > etc. Network connections are stable, no roaming :-) > > Long-term logs are for servers, so systemd doesn't want them. Right but it would be nice if it could at least forward them upstream then! Your choices are * use rsyslog, which does everything better than journald, including writing to many different databases directly, just to forward to remote host. * setup super special listener for super special journald remote log sending method that nothing but journald supports If it just supported standard, common protocols like RFC5424 flavour (that adds structured data to entries), or hell, just "send a JSONed log to that address", I could just have a bunch of my "small" devices run journald-only and send it to central logging server. But no dice, even my router needs to run extra daemon just to forward logs. Actually, having pluggable "backends" would solve a lot of that, dumb little IoT device could just have "a text file" or remote output implemented as write method and "grep the text file" as query method. Or DBI plugin that could write to SQLite and query it back. Or just remote log plugin that returned "sorry, your logs are in another castle" on query. But that's just building rsyslog but worse at that point... I wish I disliked C less than I miss those features... -- Mariusz Gronczewski (XANi) GnuPG: 0xEA8ACE64 https://devrandom.eu
Re: Journald's qualities (was: Selective rotation of journald logs)
Dnia 2024-02-23, o godz. 12:34:34 Dan Ritter napisał(a): > Stefan Monnier wrote: > > Makes one wonder why they don't use naive append-only "plain text" > > logs (tho with appropriate delimiters (maybe some kind of CSV) to > > make searches more reliable than with old-style plain text logs)? > > > > What are the advantages of journald's representation? > > I mean, to justify the slow search and large disk space usage, > > there is presumably some upside for some use cases. I can see some > > weak argument against Sqlite based on the size of Sqlite, but what > > are the advantages of journald's representation compared to a naive > > one? > > > systemd's design philosophy, observed from the outside, goes > like this: > > - assume that the machine is a laptop or mobile device that is > always changing: moves from network to network, plugs and > unplugs devices, goes to sleep and is woken up I had thought similar thoughts many times when using it. But some of the stuff that derived from that is pretty useful; for example we had a service tht downloaded encryption keys from central server to unlock the hard drive and sysV version was a bit of a mess (if service didn't ran on boot there was no unlocked volume to mount), while in systemd the dependencies allow it to automatically bring it up when mount is requested, and the mount will also be automatically done if service using it is started (if it has correct RequiresMountsFor entries). So some actually useful stuff for more "constant config" machines had flown from that > - disk space is limited, but cpu time is free In which way ? It doesn't care about wasting space, but it does assume you have blazing fast storage or else journal related ops are sluggish (and waste disk cache as it just reads hundreds of MBs to find last few lines of logfile) > - the network knows better than local config Offtopic but since Debian switched to systemd for DNS management on VPNs and suc I need to restart it sometimes multiple times to just get "right" DNS servers, because there appears to be no notion of priority: https://github.com/systemd/systemd/issues/27543 so now any time I connect to work (just openvpn tunnel, nothing fancy) I need to spam systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf few times till the dice rolls the right order of DNS servers... XANi -- Mariusz Gronczewski (XANi) GnuPG: 0xEA8ACE64 https://devrandom.eu
Re: Journald's qualities (was: Selective rotation of journald logs)
On Fri, Feb 23, 2024, 2:57 PM Dan Ritter wrote: > Stefan Monnier wrote: > > Makes one wonder why they don't use naive append-only "plain text" logs > > (tho with appropriate delimiters (maybe some kind of CSV) to make > > searches more reliable than with old-style plain text logs)? > > > > What are the advantages of journald's representation? > > I mean, to justify the slow search and large disk space usage, there is > > presumably some upside for some use cases. I can see some weak argument > > against Sqlite based on the size of Sqlite, but what are the advantages > > of journald's representation compared to a naive one? > > > systemd's design philosophy, observed from the outside, goes > like this: > bunch trimmed. Exactly correct in my view. Systemd's use-case is the desktop, not the server in the datacenter. They will be using log-aggregation software in the datacenter anyway so no use for systemd logging. We don't install desktop software on servers either, no X Windows, no gnome, etc. Network connections are stable, no roaming :-) Long-term logs are for servers, so systemd doesn't want them. > systemd thinks logs are for finding out what just happened > recently. If you wanted long-term logs, obviously you would > configure a central repository on some other machine and ship > them across the network. > > > I have nothing but praise for the Debian maintainers of rsyslog, > who have arranged it so that installing rsyslog immediately does > appropriate things to pull data out of systemd. > > -dsr- > >
Re: Journald's qualities (was: Selective rotation of journald logs)
Stefan Monnier wrote: > Makes one wonder why they don't use naive append-only "plain text" logs > (tho with appropriate delimiters (maybe some kind of CSV) to make > searches more reliable than with old-style plain text logs)? > > What are the advantages of journald's representation? > I mean, to justify the slow search and large disk space usage, there is > presumably some upside for some use cases. I can see some weak argument > against Sqlite based on the size of Sqlite, but what are the advantages > of journald's representation compared to a naive one? systemd's design philosophy, observed from the outside, goes like this: - assume that the machine is a laptop or mobile device that is always changing: moves from network to network, plugs and unplugs devices, goes to sleep and is woken up - disk space is limited, but cpu time is free - the network knows better than local config - all services serve the local user's desktop, primarily - the local user doesn't know anything but can call on a remote sysadmin - systemd is the best at doing anything, so it should do everything. If all of these assumptions match up with your particular use case, systemd is happy. Long-term logs are for servers, so systemd doesn't want them. systemd thinks logs are for finding out what just happened recently. If you wanted long-term logs, obviously you would configure a central repository on some other machine and ship them across the network. I have nothing but praise for the Debian maintainers of rsyslog, who have arranged it so that installing rsyslog immediately does appropriate things to pull data out of systemd. -dsr-
Re: Journald's qualities (was: Selective rotation of journald logs)
Dnia 2024-02-23, o godz. 10:26:49 Stefan Monnier napisał(a): > > [13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero > > bs=1M 0+15442 records in > > 0+15442 records out > > 63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s > > du -h /var/log/journal/ > > 337M/var/log/journal/44cf6f547971fc33309d1e9e02e7 > > 337M/var/log/journal/ > > > > (I've raised a bug 8 years ago about it > > https://github.com/systemd/systemd/issues/2460 ) > > Oh, that bug report is quite interesting, thanks. > Makes one wonder why they don't use naive append-only "plain text" > logs (tho with appropriate delimiters (maybe some kind of CSV) to make > searches more reliable than with old-style plain text logs)? > > What are the advantages of journald's representation? > I mean, to justify the slow search and large disk space usage, there > is presumably some upside for some use cases. The advantage of binary format is that you have separate fields for program, level etc and it is easier to filter by exactly what you want. So "systemctl status servicename" just asks log subsystem for exactly what it needs. But journald logs aren't build for querying speed, there is for example no index (in memory or on disk) that tells you where last log entries from apps are so you get pathological cases like in ticket. The problem is that you kinda do 2 types of querying on logs most often: * get me all/most of what happened in last x lines/hours * get me everything recent (or just everything period) that this particular service returned. Which kinda exclude the "simple" tricks" like "just hash over service name so at worst you have to search only 1/64th of files" as they would make the "get me everything" case slower. But maybe not by all that much. The size issue... it's appendable binary format which means that any metadata systemd adds can't be deduplicated by just having join on table with service info, it needs to be repeated every single line and I'm guessing that's where the bloat comes from. I rambled about SQLite as it is kinda closest comparison for embedded binary format that can be searched and I think that being able to run SQL queries (sqlite now even have full text search) against your local logs with no database daemons running would be killer feature on some of the smaller devices. > argument against Sqlite based on the size of Sqlite, I'd argue that's not really technically relevant as in most cases you'd have other app in system using sqlite for *something* and will have installed the library anyway. And space savings gets wasted just on how verbose journald format is. > but what are the > advantages of journald's representation compared to a naive one? in short: querability without text parsing. That's about it. Which *is* a great feature, do not get me wrong, I use it all the time, but the slowdown caused by its unfortunate implementation detail is also showing up when I use it pretty often as the most common use case is "something broke, log in on server that has been running on automation for last 3 months and see what is happening" It does have some features around FSS (signing logs) so for folks that need it there isn't many alternatives as it would be hard to make similar feature on database. But on flipside inability to separate log into log rotation groups is *nasty*. You basically can't have verbose service in the system as that will rotate away any old logs of other services away. And looking for those logs is also the worst case: [17:57:56]cthulhu:~☠ echo 3 > /proc/sys/vm/drop_caches [17:58:14]cthulhu:~☠ time systemctl status nginx |grep Notice Notice: journal has been rotated since unit was started, output may be incomplete. real0m1,645s user0m0,000s sys 0m0,094s [17:58:19]cthulhu:~☠ echo 3 > /proc/sys/vm/drop_caches [17:58:28]cthulhu:~☠ time sqlite /tmp/log_indexed.sqlite 'select * from log where source = "nginx";' >/dev/null real0m0,374s user0m0,000s sys 0m0,009s I guess another one being appendable log is that in case of crash it's easier to recover than embedded database... ...or so would I say if I didn't saw journald throwing away binary logs with some errors after crash. > > > Stefan > > > PS: FWIW, the only time I thought about the design of a log format, > I was thinking along the lines of reducing redundancy, i.e. keeping > a kind of "suffix tree" of past messages so as to replace the many > repeated chunks of text with references to past log entries. > Then I realized that I could get the same result much more simply by > gzipping the files, which would naturally be done as part of > log rotation 🙂 > One-service-per-file approach is honestly good enough for most stuff. PITA to get chronological order thought, every approach really have some drawbacks and benefits. But honestly ? Send it to central logger and worry there if you have a chance, especially if you do some parsing there can be
Journald's qualities (was: Selective rotation of journald logs)
> [13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero bs=1M > 0+15442 records in > 0+15442 records out > 63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s > du -h /var/log/journal/ > 337M /var/log/journal/44cf6f547971fc33309d1e9e02e7 > 337M /var/log/journal/ > > (I've raised a bug 8 years ago about it > https://github.com/systemd/systemd/issues/2460 ) Oh, that bug report is quite interesting, thanks. Makes one wonder why they don't use naive append-only "plain text" logs (tho with appropriate delimiters (maybe some kind of CSV) to make searches more reliable than with old-style plain text logs)? What are the advantages of journald's representation? I mean, to justify the slow search and large disk space usage, there is presumably some upside for some use cases. I can see some weak argument against Sqlite based on the size of Sqlite, but what are the advantages of journald's representation compared to a naive one? Stefan PS: FWIW, the only time I thought about the design of a log format, I was thinking along the lines of reducing redundancy, i.e. keeping a kind of "suffix tree" of past messages so as to replace the many repeated chunks of text with references to past log entries. Then I realized that I could get the same result much more simply by gzipping the files, which would naturally be done as part of log rotation 🙂
