Package: keepalived
Version: 1:2.0.10-1
Severity: critical
Tags: upstream
Justification: breaks the whole system
Dear Maintainer,
*** Reporter, please consider answering these questions, where appropriate ***
* What led up to the situation? A bug in keepalived with lists made the
whole server go down. Because keepalived kept crashing forever, an insane
amount of logs was generated. More than 30 GB of logs were generated in about
90 min, filling up the disk very rapidly.
Because of this, the software of the whole server went down and was
affected. In the end a large application suffered severely from this.
* What exactly did you do (or not do) that was effective (or
ineffective)? We cleaned up the logs and restarted all services
* What was the outcome of this action? Server is running again, however
there is a large chanche this will occur again sooner or later.
* What outcome did you expect instead? At least no crashing forever of
keepalived. Maybe improve systemd unit file it cannot restart forever but stops
after x tries (Up to Debian maintainer)
I attached relevant logs here, as you can see mb's per second of logs are
generated, this until disk was completely full:
/var/log/daemon.log
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12803
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12803) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12804
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12804) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12805
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12805) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12806
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12806) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12807
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12807) died:
Respawning
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12808
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12808) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12809
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12809) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12810
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12810) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12811
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12811) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12814
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12814) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12816
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12816) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12817
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12817) died:
Respawning
Dec 22 00:00:01 db1 Keepalived[19081]: Starting VRRP child process, pid=12818
Dec 22 00:00:01 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:01 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:01 db1 Keepalived[19081]: VRRP child process(12818) died:
Respawning
TRUNCATED HERE
/var/log/syslog
Dec 22 00:00:00 db1 Keepalived_vrrp[12242]: Registering Kernel netlink reflector
Dec 22 00:00:00 db1 Keepalived_vrrp[12242]: Registering Kernel netlink command
channel
Dec 22 00:00:00 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:00 db1 Keepalived_vrrp[12242]: Opening file
'/etc/keepalived/keepalived.conf'.
Dec 22 00:00:00 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:00 db1 Keepalived_vrrp[12242]: Opening file
'/etc/keepalived/conf.d/global.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12242]: Opening file
'/etc/keepalived/conf.d/mysql.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12242]: Opening file
'/etc/keepalived/conf.d/redis.conf'.
Dec 22 00:00:00 db1 rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0"
x-pid="537" x-info="https://www.rsyslog.com";] rsyslogd was HUPed
Dec 22 00:00:00 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:00 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:00 db1 Keepalived[19081]: VRRP child process(12242) died:
Respawning
Dec 22 00:00:00 db1 Keepalived[19081]: Starting VRRP child process, pid=12243
Dec 22 00:00:00 db1 Keepalived_vrrp[12243]: Registering Kernel netlink reflector
Dec 22 00:00:00 db1 Keepalived_vrrp[12243]: Registering Kernel netlink command
channel
Dec 22 00:00:00 db1 Keepalived_vrrp[12243]: Opening file
'/etc/keepalived/keepalived.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12243]: Opening file
'/etc/keepalived/conf.d/global.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12243]: Opening file
'/etc/keepalived/conf.d/mysql.conf'.
Dec 22 00:00:00 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:00 db1 Keepalived_vrrp[12243]: Opening file
'/etc/keepalived/conf.d/redis.conf'.
Dec 22 00:00:00 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:00 db1 Keepalived[19081]: VRRP child process(12243) died:
Respawning
Dec 22 00:00:00 db1 Keepalived[19081]: Starting VRRP child process, pid=12244
Dec 22 00:00:00 db1 Keepalived_vrrp[12244]: Registering Kernel netlink reflector
Dec 22 00:00:00 db1 Keepalived_vrrp[12244]: Registering Kernel netlink command
channel
Dec 22 00:00:00 db1 Keepalived_vrrp[12244]: Opening file
'/etc/keepalived/keepalived.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12244]: Opening file
'/etc/keepalived/conf.d/global.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12244]: Opening file
'/etc/keepalived/conf.d/mysql.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12244]: Opening file
'/etc/keepalived/conf.d/redis.conf'.
Dec 22 00:00:00 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:00 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:00 db1 Keepalived[19081]: VRRP child process(12244) died:
Respawning
Dec 22 00:00:00 db1 Keepalived[19081]: Starting VRRP child process, pid=12246
Dec 22 00:00:00 db1 Keepalived_vrrp[12246]: Registering Kernel netlink reflector
Dec 22 00:00:00 db1 Keepalived_vrrp[12246]: Registering Kernel netlink command
channel
Dec 22 00:00:00 db1 Keepalived_vrrp[12246]: Opening file
'/etc/keepalived/keepalived.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12246]: Opening file
'/etc/keepalived/conf.d/global.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12246]: Opening file
'/etc/keepalived/conf.d/mysql.conf'.
Dec 22 00:00:00 db1 Keepalived_vrrp[12246]: Opening file
'/etc/keepalived/conf.d/redis.conf'.
Dec 22 00:00:00 db1 keepalived[19081]: malloc(): smallbin double linked list
corrupted
Dec 22 00:00:00 db1 Keepalived[19081]: Keepalived_vrrp exited due to signal 6
Dec 22 00:00:00 db1 Keepalived[19081]: VRRP child process(12246) died:
Respawning
TRUNCATED HERE
-- System Information:
Debian Release: 10.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-6-cloud-amd64 (SMP w/6 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages keepalived depends on:
ii iproute2 4.20.0-2
ii libc6 2.28-10
ii libglib2.0-0 2.58.3-2+deb10u2
ii libip4tc0 1.8.2-4
ii libip6tc0 1.8.2-4
ii libjson-c3 0.12.1+ds-2
ii libnl-3-200 3.4.0-1
ii libnl-genl-3-200 3.4.0-1
ii libsnmp30 5.7.3+dfsg-5
ii libssl1.1 1.1.1d-0+deb10u2
ii libxtables12 1.8.2-4
Versions of packages keepalived recommends:
ii ipvsadm 1:1.29-1
keepalived suggests no packages.
-- no debconf information
