** Description changed: [Impact] In recent versions of rsyslog and librelp, the imrelp module leaks file descriptors due to a bug where it does not correctly close sockets, and instead, leaves them in the CLOSE_WAIT state. This causes rsyslogd on busy servers to eventually hit the limit of maximum open files allowed, which locks rsyslogd up until it is restarted. A workaround is to restart rsyslogd every month or so to manually close all of the open sockets. Only users of the imrelp module are affected, and not rsyslog users in general. [Testcase] Install the rsyslog-relp module like so: $ sudo apt install rsyslog rsyslog-relp Next, generate a working directory, and make a config file that loads the relp module. $ sudo mkdir /workdir $ cat << EOF >> ./spool.conf \$LocalHostName spool \$AbortOnUncleanConfig on \$PreserveFQDN on global( workDirectory="/workdir" maxMessageSize="256k" ) main_queue(queue.type="Direct") module(load="imrelp") input( type="imrelp" name="imrelp" port="601" ruleset="spool" MaxDataSize="256k" ) ruleset(name="spool" queue.type="direct") { } # Just so rsyslog doesn't whine that we do not have outputs ruleset(name="noop" queue.type="direct") { action( type="omfile" name="omfile" file="/workdir/spool.log" ) } EOF Verify that the config is valid, then start a rsyslog server. $ sudo rsyslogd -f ./spool.conf -N9 $ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid Fetch the rsyslogd PID and check for open files. $ RLOGPID=$(cat /workdir/rsyslogd.pid) $ sudo ls -l /proc/$RLOGPID/fd total 0 lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]' lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]' lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]' lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]' We have 3 sockets open by default. Next, use netcat to open 100 connections: $ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done Now check for open file descriptors, and there will be an extra 100 sockets in the list: $ sudo ls -l /proc/$RLOGPID/fd https://paste.ubuntu.com/p/f6NQVNbZcR/ We can check the state of these sockets with: $ ss -t https://paste.ubuntu.com/p/7Ts2FbxJrg/ The listening sockets will be in CLOSE-WAIT, and the netcat sockets will be in FIN-WAIT-2. + $ ss -t | grep CLOSE-WAIT | wc -l + 100 + If you install the test package available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test When you open connections with netcat, these will be closed properly, and the file descriptor leak will be fixed. [Where problems could occur] If a regression were to occur, it would be limited to users of the imrelp module, which is a part of the rsyslogd-relp package, and depends on librelp. rsyslog-relp is not part of a default installation of rsyslog, and is opt in by changing a configuration file to enable imrelp. The changes to rsyslog implement a testcase which exercises the problematic code to ensure things are working as expected, and should run during autopkgtest time. [Other] Upstream bug list: https://github.com/rsyslog/rsyslog/issues/4350 https://github.com/rsyslog/rsyslog/issues/4005 https://github.com/rsyslog/librelp/issues/188 https://github.com/rsyslog/librelp/pull/193 The following commits fix the problem: rsyslogd ======== commit baee0bd5420649329793746f0daf87c4f59fe6a6 Author: Andre lorbach <alorb...@adiscon.com> Date: Thu Apr 9 13:00:35 2020 +0200 Subject: testbench: Add test for imrelp to check broken session handling. Link: https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6 librelp ======= commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0 Author: Andre lorbach <alorb...@adiscon.com> Date: Mon May 11 14:59:55 2020 +0200 Subject: fix memory leak on session break. Link: https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0 commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a Author: Andre lorbach <alorb...@adiscon.com> Date: Wed Apr 8 15:55:32 2020 +0200 Subject: replsess: fix double free of sendbuf in some cases. Link: https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a commit 3797944fb62273fa1164acd3104f0894b337c4d0 Author: Ognyan Kulev <ogn...@ognyankulev.com> Date: Mon Jun 15 14:10:08 2020 +0300 Subject: Fix FD leak when socket shutdown is one-sided Link: https://github.com/rsyslog/librelp/commit/3797944fb62273fa1164acd3104f0894b337c4d0
-- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/1908473 Title: rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which leads to file descriptor leak Status in librelp package in Ubuntu: In Progress Status in rsyslog package in Ubuntu: Fix Released Status in librelp source package in Focal: In Progress Status in rsyslog source package in Focal: Won't Fix Status in librelp source package in Groovy: In Progress Status in rsyslog source package in Groovy: Fix Released Status in librelp source package in Hirsute: In Progress Status in rsyslog source package in Hirsute: Fix Released Bug description: [Impact] In recent versions of rsyslog and librelp, the imrelp module leaks file descriptors due to a bug where it does not correctly close sockets, and instead, leaves them in the CLOSE_WAIT state. This causes rsyslogd on busy servers to eventually hit the limit of maximum open files allowed, which locks rsyslogd up until it is restarted. A workaround is to restart rsyslogd every month or so to manually close all of the open sockets. Only users of the imrelp module are affected, and not rsyslog users in general. [Testcase] Install the rsyslog-relp module like so: $ sudo apt install rsyslog rsyslog-relp Next, generate a working directory, and make a config file that loads the relp module. $ sudo mkdir /workdir $ cat << EOF >> ./spool.conf \$LocalHostName spool \$AbortOnUncleanConfig on \$PreserveFQDN on global( workDirectory="/workdir" maxMessageSize="256k" ) main_queue(queue.type="Direct") module(load="imrelp") input( type="imrelp" name="imrelp" port="601" ruleset="spool" MaxDataSize="256k" ) ruleset(name="spool" queue.type="direct") { } # Just so rsyslog doesn't whine that we do not have outputs ruleset(name="noop" queue.type="direct") { action( type="omfile" name="omfile" file="/workdir/spool.log" ) } EOF Verify that the config is valid, then start a rsyslog server. $ sudo rsyslogd -f ./spool.conf -N9 $ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid Fetch the rsyslogd PID and check for open files. $ RLOGPID=$(cat /workdir/rsyslogd.pid) $ sudo ls -l /proc/$RLOGPID/fd total 0 lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]' lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]' lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]' lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]' We have 3 sockets open by default. Next, use netcat to open 100 connections: $ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done Now check for open file descriptors, and there will be an extra 100 sockets in the list: $ sudo ls -l /proc/$RLOGPID/fd https://paste.ubuntu.com/p/f6NQVNbZcR/ We can check the state of these sockets with: $ ss -t https://paste.ubuntu.com/p/7Ts2FbxJrg/ The listening sockets will be in CLOSE-WAIT, and the netcat sockets will be in FIN-WAIT-2. $ ss -t | grep CLOSE-WAIT | wc -l 100 If you install the test package available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test When you open connections with netcat, these will be closed properly, and the file descriptor leak will be fixed. [Where problems could occur] If a regression were to occur, it would be limited to users of the imrelp module, which is a part of the rsyslogd-relp package, and depends on librelp. rsyslog-relp is not part of a default installation of rsyslog, and is opt in by changing a configuration file to enable imrelp. The changes to rsyslog implement a testcase which exercises the problematic code to ensure things are working as expected, and should run during autopkgtest time. [Other] Upstream bug list: https://github.com/rsyslog/rsyslog/issues/4350 https://github.com/rsyslog/rsyslog/issues/4005 https://github.com/rsyslog/librelp/issues/188 https://github.com/rsyslog/librelp/pull/193 The following commits fix the problem: rsyslogd ======== commit baee0bd5420649329793746f0daf87c4f59fe6a6 Author: Andre lorbach <alorb...@adiscon.com> Date: Thu Apr 9 13:00:35 2020 +0200 Subject: testbench: Add test for imrelp to check broken session handling. Link: https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6 librelp ======= commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0 Author: Andre lorbach <alorb...@adiscon.com> Date: Mon May 11 14:59:55 2020 +0200 Subject: fix memory leak on session break. Link: https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0 commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a Author: Andre lorbach <alorb...@adiscon.com> Date: Wed Apr 8 15:55:32 2020 +0200 Subject: replsess: fix double free of sendbuf in some cases. Link: https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a commit 3797944fb62273fa1164acd3104f0894b337c4d0 Author: Ognyan Kulev <ogn...@ognyankulev.com> Date: Mon Jun 15 14:10:08 2020 +0300 Subject: Fix FD leak when socket shutdown is one-sided Link: https://github.com/rsyslog/librelp/commit/3797944fb62273fa1164acd3104f0894b337c4d0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/librelp/+bug/1908473/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp