Bug#1070281: logcheck: becomes less and less usable because of user-level logs
On Fri, 3 May 2024, 12:44 Francesco Potortì, wrote: > > > One cure would be to have logcheck ignore user-level messages, and only > care about system-level ones. Is that possible? > > > >maybe it is possible - how do you define "system-level message"? > > Those created by root-owned processes, that would be a good start. I > definitely care about Sshd messages, much less about Gvfsd ones, and even > less by those generated by Telegram running over Snapd. For some reason, > the problem has vastly increased after the advent of systemctl. The options seem to be 1.you make a local rule that ignores all messages from known culprets -- so you might jusy want to do a version of "^timestamp hostname (Telegram|gvfsd)". This works today, but does need you to know what you want to ignore 2.you tell logcheck to.not check the journal at all - also possoble today: simply remove "journal" from the file in /etc/logcheck/logcheck.logfiles.d (i dont know if this is that helpful!) 3. i have work in progress to allow you to tell logcheck to only check a subset of the journal by passif arguments to journalctl. Looking at the journalctl.man-page: --unit ssh.service will only show messages from ssh eg --system might exclude things like telegram (untested!) eg --priority might also be helpful eg _UID=0 might select only things run by root (but that would probably exclude things run by special users like apache) eg --priority might also help? This needs a small change in logcheck to make JOURNALCTL_OPTS settable from the config file - this is WiP already! (logcheck currently hardcoded this to an empty array) other thoughts: - we could definitely make logcheck only report the first N lines. I can broadly see how to implement this. you can almost do this today by making a "syslog-summary" script!
Bug#1070281: logcheck: becomes less and less usable because of user-level logs
>logcheck-database was mostly dormant sround that time. im hoping to improve >that, but it is a big task and needs some wider >improvements. So: please bear with it! I really appreciate your work, thanks. > however: >- a bug in a daemon should ideally be reported and fixed in the daemon Sure. And I routinely do. But I use logcheck to warn me of serious problems at the system level. Detecting user-level daemon bugs is not my priority when using logcheck. Especially because when it happens, the deluge of info hides the possible serious system-level problems that I am mostly interested into. >- this may include logging "too much" -- i would suggest discussing with >upstream as they may be open to improvements In the last two years I observed and reported many of those bugs, before hiding them behind a custom logcheck rule. Most were acknowledged, some were fixed. >- you didnt give any examples so not sure how anyone can help you I can find the relevant bug reports, but that's not the issue I am raising. When a random bug in gvfsd (just to mention the latest one) risks filling my root partition with multi-GB logs, and logcheck sends me hundred-MB mails full of useless stuff, that makes logcheck useless. In the past, when I detected a bug, I kept the email for later, to report the bug when I found the time. Now it happens way too often for me and I am seriously considering shutting it off. Reporting bugs is voluntary work which I gladly do in my free time. But if this subtracts work time I cannot afford it. > I had many email tens of megabytes long. > >(there's already a request to split the report if it is long) This is not the problem for me. Logcheck should stop logging after a configurable number of lines (for me, that would be around 100, certainly no more than 1000), because in my experience that just indicates a bug in the logging procedure of some daemon or some missing logcheck filters, and I lose my log anyway, as I do not have the time to sift through the mostly useless reported stuff. Additionally, user-level reports should be separated from system-level ones. I am not knowledgeable enough to know how to do that if not by single crafted local rules, but if I cannot have that I will give up with logcheck. Again, I am interested in system-level serious problems, and anything that obfuscates that makes logcheck useless and worse. >If.you wanted to chamge the world, get upstream authors to agree some standard >where messges are easier to identify as routine >and then logcheck could more easily ignore that. i wont hold my breath >for thay Yeah, I feared that the answer would be similar :( > One cure would be to have logcheck ignore user-level messages, and only care > about system-level ones. Is that possible? > >maybe it is possible - how do you define "system-level message"? Those created by root-owned processes, that would be a good start. I definitely care about Sshd messages, much less about Gvfsd ones, and even less by those generated by Telegram running over Snapd. For some reason, the problem has vastly increased after the advent of systemctl. Again, thank you for your work, that's very much appreciated. And again, I know these problems have always existed, but for some reason they have increased a lot and they keep increasing. Sincerely -- Francesco Potortì (ricercatore)Mobile: +39.348.8283.107 ISTI - CNR, Pisa, ItalyWeb:http://fly.isti.cnr.it
Bug#1070281: logcheck: becomes less and less usable because of user-level logs
control: reassign -1 logcheck-database thanks (this is mostly about logcheck-database) On Fri, 3 May 2024, 09:39 Francesco Potortì, wrote: > > > Starting maybe a couple years ago, logcheck spits an amount of stuff that > has now become unamnageable. logcheck-database was mostly dormant sround that time. im hoping to improve that, but it is a big task and needs some wider improvements. So: please bear with it! While in the beginning logs were relevant to the system, which is what I > want, now any bug in a user-level daemon writing info to the syslog or to > the systemsctl thing makes havoc. i sympathise however: - a bug in a daemon should ideally be reported and fixed in the daemon - this may include logging "too much" -- i would suggest discussing with upstream as they may be open to improvements - you didnt give any examples so not sure how anyone can help you I had many email tens of megabytes long. (there's already a request to split the report if it is long) In the last month, reports have been completely useless to me as they are > filled with useless things and I am busy adding local rules to remove them > > Now I have reached a point where its usefulness is negative: I spend more > time adding rules than it is worth for me getting the info. > i sympathise - but writing local rules is always going to be needed. i think we can do much better than what we have, but realistically it is hard to do in this way. If.you wanted to chamge the world, get upstream authors to agree some standard where messges are easier to identify as routine and then logcheck could more easily ignore that. i wont hold my breath for thay > One cure would be to have logcheck ignore user-level messages, and only > care about system-level ones. Is that possible? > maybe it is possible - how do you define "system-level message"?
Bug#1070281: logcheck: becomes less and less usable because of user-level logs
Package: logcheck Version: 1.4.3 Severity: normal X-Debbugs-Cc: none, Francesco Potortì I have used logcheck on testing for many years. Being on testing, I occasionally had to add local rules to keep it up. Usually logcheck signaled few lines every day, sometimes no line, sometims (at reboot or big updates) many lines. Starting maybe a couple years ago, logcheck spits an amount of stuff that has now become unamnageable. While in the beginning logs were relevant to the system, which is what I want, now any bug in a user-level daemon writing info to the syslog or to the systemsctl thing makes havoc. I had many email tens of megabytes long. In the last month, reports have been completely useless to me as they are filled with useless things and I am busy adding local rules to remove them Now I have reached a point where its usefulness is negative: I spend more time adding rules than it is worth for me getting the info. One cure would be to have logcheck ignore user-level messages, and only care about system-level ones. Is that possible? -- Francesco Potortì (ricercatore)Mobile: +39.348.8283.107 ISTI - CNR, Pisa, ItalyWeb:http://fly.isti.cnr.it -- System Information: Debian Release: trixie/sid APT prefers testing APT policy: (990, 'testing'), (101, 'unstable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 6.6.15-amd64 (SMP w/24 CPU threads; PREEMPT) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=C.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages logcheck depends on: ii adduser3.137 hi cron [cron-daemon] 3.0pl1-189 ii exim4-daemon-light [mail-transport-agent] 4.97-5 ii lockfile-progs 0.1.19+nmu1 ii logtail1.4.3 ii mime-construct 1.12+really1.11-1 Versions of packages logcheck recommends: ii logcheck-database 1.4.3 Versions of packages logcheck suggests: ii rsyslog [system-log-daemon] 8.2402.0-1 -- Configuration Files: /etc/cron.d/logcheck changed: PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root @reboot logcheckif [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R; fi 2 4 * * * logcheckif [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi /etc/logcheck/header.txt [Errno 13] Permission denied: '/etc/logcheck/header.txt' /etc/logcheck/logcheck.conf [Errno 13] Permission denied: '/etc/logcheck/logcheck.conf' /etc/logcheck/logcheck.logfiles [Errno 13] Permission denied: '/etc/logcheck/logcheck.logfiles' /etc/logcheck/logcheck.logfiles.d/journal.logfiles [Errno 13] Permission denied: '/etc/logcheck/logcheck.logfiles.d/journal.logfiles' /etc/logcheck/logcheck.logfiles.d/syslog.logfiles [Errno 13] Permission denied: '/etc/logcheck/logcheck.logfiles.d/syslog.logfiles' -- no debconf information
