Package: syslog-ng Version: 3.28.1-2 Severity: important Tags: upstream I sincerely apologise since this is not a good bugreport, so if you really feel like completely clueless you can close it, but the problem will probably presist.
The environments use SysV init (none of the systems are systemd-infected). (It is possible that it have happened on a systemd-infected server, too, but I am not sure now as it was many weeks before and back then I was not aware about the cause of the problem.) What happens: * usually around 03:26 (cron.daily / logrotate) syslog-ng gets stuck on a writev() call, which is supposed to write the actual incoming log line into a physical file on the system. strace shows that syslog-ng gets signals (like HUP or TERM), handles them and ignores them and go back to writev(). * during this time the syslog service blocks (after a while). * this in turn blocks everything using syslog service, and I mean everything. Notable mentions are dhcp server, vpn server, various cron services. (Blocking cron may also mean system load goes sky high due to many cron spawns running at once.) * the disk is not full. * HUP, TERM and normal signals get ignored. ILL or KILL kills the daemon and the system starts working again as syslog-ng gets respawned. Log rotation is done using 'invoke-rc.d syslog-ng reload' which uses `start-stop-daemon` to send SIGNAL1 (SIGHUP) then uses `syslog-ng-ctl stats` to see when the daemon reports back, which never happens: the call gets stuck as well. [I would guess SIGHUP handler runs amok.] This seems to happen in 3.19.1-5 and 3.28.1-2 as well, but only on the fraction of the servers, but there it is recurring around weekly. Nothing relevant is in dmesg or elsewhere. :-( -- System Information: Debian Release: bullseye/sid APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.8.6 (SMP w/8 CPU threads) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_FIRMWARE_WORKAROUND, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF8, LC_CTYPE=en_US.UTF8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /bin/bash Init: sysvinit (via /sbin/init) LSM: AppArmor: enabled Versions of packages syslog-ng depends on: ii syslog-ng-core 3.28.1-2+b1 ii syslog-ng-mod-mongodb 3.28.1-2+b1 ii syslog-ng-mod-sql 3.28.1-2+b1 Versions of packages syslog-ng recommends: pn syslog-ng-mod-add-contextual-data <none> pn syslog-ng-mod-amqp <none> pn syslog-ng-mod-examples <none> pn syslog-ng-mod-extra <none> pn syslog-ng-mod-geoip2 <none> pn syslog-ng-mod-getent <none> pn syslog-ng-mod-graphite <none> pn syslog-ng-mod-http <none> pn syslog-ng-mod-map-value-pairs <none> pn syslog-ng-mod-python <none> pn syslog-ng-mod-rdkafka <none> pn syslog-ng-mod-redis <none> pn syslog-ng-mod-riemann <none> pn syslog-ng-mod-slog <none> pn syslog-ng-mod-smtp <none> pn syslog-ng-mod-snmp <none> pn syslog-ng-mod-stardate <none> pn syslog-ng-mod-stomp <none> pn syslog-ng-mod-xml-parser <none> syslog-ng suggests no packages. -- no debconf information