Public bug reported:
[Impact]
In recent versions of rsyslog and librelp, the imrelp module leaks file
descriptors due to a bug where it does not correctly close sockets, and
instead, leaves them in the CLOSE_WAIT state.
This causes rsyslogd on busy servers to eventually hit the limit of
maximum open files allowed, which locks rsyslogd up until it is
restarted.
A workaround is to restart rsyslogd every month or so to manually close
all of the open sockets.
Only users of the imrelp module are affected, and not rsyslog users in
general.
[Testcase]
Install the rsyslog-relp module like so:
$ sudo apt install rsyslog rsyslog-relp
Next, generate a working directory, and make a config file that loads
the relp module.
$ sudo mkdir /workdir
$ cat << EOF >> ./spool.conf
\$LocalHostName spool
\$AbortOnUncleanConfig on
\$PreserveFQDN on
global(
workDirectory="/workdir"
maxMessageSize="256k"
)
main_queue(queue.type="Direct")
module(load="imrelp")
input(
type="imrelp"
name="imrelp"
port="601"
ruleset="spool"
MaxDataSize="256k"
)
ruleset(name="spool" queue.type="direct") {
}
# Just so rsyslog doesn't whine that we do not have outputs
ruleset(name="noop" queue.type="direct") {
action(
type="omfile"
name="omfile"
file="/workdir/spool.log"
)
}
EOF
Verify that the config is valid, then start a rsyslog server.
$ sudo rsyslogd -f ./spool.conf -N9
$ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid
Fetch the rsyslogd PID and check for open files.
$ RLOGPID=$(cat /workdir/rsyslogd.pid)
$ sudo ls -l /proc/$RLOGPID/fd
total 0
lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'
We have 3 sockets open by default. Next, use netcat to open 100
connections:
$ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done
Now check for open file descriptors, and there will be an extra 100 sockets
in the list:
$ sudo ls -l /proc/$RLOGPID/fd
https://paste.ubuntu.com/p/f6NQVNbZcR/
We can check the state of these sockets with:
$ ss -t
https://paste.ubuntu.com/p/7Ts2FbxJrg/
The listening sockets will be in CLOSE-WAIT, and the netcat sockets will
be in FIN-WAIT-2.
If you install the test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test
When you open connections with netcat, these will be closed properly,
and the file descriptor leak will be fixed.
[Where problems could occur]
If a regression were to occur, it would be limited to users of the
imrelp module, which is a part of the rsyslogd-relp package, and depends
on librelp.
rsyslog-relp is not part of a default installation of rsyslog, and is
opt in by changing a configuration file to enable imrelp.
The changes to rsyslog implement a testcase which exercises the
problematic code to ensure things are working as expected, and should
run during autopkgtest time.
[Other]
Upstream bug list:
https://github.com/rsyslog/rsyslog/issues/4350
https://github.com/rsyslog/rsyslog/issues/4005
https://github.com/rsyslog/librelp/issues/188
The following commits fix the problem:
rsyslogd
========
commit baee0bd5420649329793746f0daf87c4f59fe6a6
Author: Andre lorbach <alorb...@adiscon.com>
Date: Thu Apr 9 13:00:35 2020 +0200
Subject: testbench: Add test for imrelp to check broken session handling.
Link:
https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6
librelp
=======
commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
Author: Andre lorbach <alorb...@adiscon.com>
Date: Mon May 11 14:59:55 2020 +0200
Subject: fix memory leak on session break.
Link:
https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a
Author: Andre lorbach <alorb...@adiscon.com>
Date: Wed Apr 8 15:55:32 2020 +0200
Subject: replsess: fix double free of sendbuf in some cases.
Link:
https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a
** Affects: librelp (Ubuntu)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: rsyslog (Ubuntu)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: librelp (Ubuntu Focal)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: rsyslog (Ubuntu Focal)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: librelp (Ubuntu Groovy)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: rsyslog (Ubuntu Groovy)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: librelp (Ubuntu Hirsute)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: rsyslog (Ubuntu Hirsute)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Tags: sts
** Also affects: rsyslog (Ubuntu Hirsute)
Importance: Undecided
Status: New
** Also affects: rsyslog (Ubuntu Focal)
Importance: Undecided
Status: New
** Also affects: rsyslog (Ubuntu Groovy)
Importance: Undecided
Status: New
** Also affects: librelp (Ubuntu)
Importance: Undecided
Status: New
** Changed in: librelp (Ubuntu Focal)
Status: New => In Progress
** Changed in: librelp (Ubuntu Groovy)
Status: New => In Progress
** Changed in: librelp (Ubuntu Hirsute)
Status: New => In Progress
** Changed in: rsyslog (Ubuntu Focal)
Status: New => In Progress
** Changed in: rsyslog (Ubuntu Groovy)
Status: New => In Progress
** Changed in: rsyslog (Ubuntu Hirsute)
Status: New => In Progress
** Changed in: librelp (Ubuntu Focal)
Importance: Undecided => Medium
** Changed in: librelp (Ubuntu Groovy)
Importance: Undecided => Medium
** Changed in: librelp (Ubuntu Hirsute)
Importance: Undecided => Medium
** Changed in: rsyslog (Ubuntu Focal)
Importance: Undecided => Medium
** Changed in: rsyslog (Ubuntu Groovy)
Importance: Undecided => Medium
** Changed in: rsyslog (Ubuntu Hirsute)
Importance: Undecided => Medium
** Changed in: librelp (Ubuntu Focal)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Changed in: librelp (Ubuntu Groovy)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Changed in: librelp (Ubuntu Hirsute)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Changed in: rsyslog (Ubuntu Focal)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Changed in: rsyslog (Ubuntu Groovy)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Changed in: rsyslog (Ubuntu Hirsute)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Tags added: sts
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1908473
Title:
rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
leads to file descriptor leak
Status in librelp package in Ubuntu:
In Progress
Status in rsyslog package in Ubuntu:
In Progress
Status in librelp source package in Focal:
In Progress
Status in rsyslog source package in Focal:
In Progress
Status in librelp source package in Groovy:
In Progress
Status in rsyslog source package in Groovy:
In Progress
Status in librelp source package in Hirsute:
In Progress
Status in rsyslog source package in Hirsute:
In Progress
Bug description:
[Impact]
In recent versions of rsyslog and librelp, the imrelp module leaks
file descriptors due to a bug where it does not correctly close
sockets, and instead, leaves them in the CLOSE_WAIT state.
This causes rsyslogd on busy servers to eventually hit the limit of
maximum open files allowed, which locks rsyslogd up until it is
restarted.
A workaround is to restart rsyslogd every month or so to manually
close all of the open sockets.
Only users of the imrelp module are affected, and not rsyslog users in
general.
[Testcase]
Install the rsyslog-relp module like so:
$ sudo apt install rsyslog rsyslog-relp
Next, generate a working directory, and make a config file that loads
the relp module.
$ sudo mkdir /workdir
$ cat << EOF >> ./spool.conf
\$LocalHostName spool
\$AbortOnUncleanConfig on
\$PreserveFQDN on
global(
workDirectory="/workdir"
maxMessageSize="256k"
)
main_queue(queue.type="Direct")
module(load="imrelp")
input(
type="imrelp"
name="imrelp"
port="601"
ruleset="spool"
MaxDataSize="256k"
)
ruleset(name="spool" queue.type="direct") {
}
# Just so rsyslog doesn't whine that we do not have outputs
ruleset(name="noop" queue.type="direct") {
action(
type="omfile"
name="omfile"
file="/workdir/spool.log"
)
}
EOF
Verify that the config is valid, then start a rsyslog server.
$ sudo rsyslogd -f ./spool.conf -N9
$ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid
Fetch the rsyslogd PID and check for open files.
$ RLOGPID=$(cat /workdir/rsyslogd.pid)
$ sudo ls -l /proc/$RLOGPID/fd
total 0
lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'
We have 3 sockets open by default. Next, use netcat to open 100
connections:
$ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done
Now check for open file descriptors, and there will be an extra 100 sockets
in the list:
$ sudo ls -l /proc/$RLOGPID/fd
https://paste.ubuntu.com/p/f6NQVNbZcR/
We can check the state of these sockets with:
$ ss -t
https://paste.ubuntu.com/p/7Ts2FbxJrg/
The listening sockets will be in CLOSE-WAIT, and the netcat sockets
will be in FIN-WAIT-2.
If you install the test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test
When you open connections with netcat, these will be closed properly,
and the file descriptor leak will be fixed.
[Where problems could occur]
If a regression were to occur, it would be limited to users of the
imrelp module, which is a part of the rsyslogd-relp package, and
depends on librelp.
rsyslog-relp is not part of a default installation of rsyslog, and is
opt in by changing a configuration file to enable imrelp.
The changes to rsyslog implement a testcase which exercises the
problematic code to ensure things are working as expected, and should
run during autopkgtest time.
[Other]
Upstream bug list:
https://github.com/rsyslog/rsyslog/issues/4350
https://github.com/rsyslog/rsyslog/issues/4005
https://github.com/rsyslog/librelp/issues/188
The following commits fix the problem:
rsyslogd
========
commit baee0bd5420649329793746f0daf87c4f59fe6a6
Author: Andre lorbach <alorb...@adiscon.com>
Date: Thu Apr 9 13:00:35 2020 +0200
Subject: testbench: Add test for imrelp to check broken session handling.
Link:
https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6
librelp
=======
commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
Author: Andre lorbach <alorb...@adiscon.com>
Date: Mon May 11 14:59:55 2020 +0200
Subject: fix memory leak on session break.
Link:
https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a
Author: Andre lorbach <alorb...@adiscon.com>
Date: Wed Apr 8 15:55:32 2020 +0200
Subject: replsess: fix double free of sendbuf in some cases.
Link:
https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/librelp/+bug/1908473/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
