Serious regression in systemd 215-17+deb8u10
Hello, We have discovered that the latest version of systemd uploaded to jessie is causing systemd-journald to use an ever increasing amount of memory, eventually leading to all available memory being consumed. This has been observed on multiple different systems we use which are logging quite heavily and have upgraded to the latest systemd package. We have some systems which haven't had the new security patches applied and are not observing this behaviour - there is a clear correlation with the latest version of the systemd package. It seems quite similar to situation described on Debian bug 920018 and hope that a fix can be found. thanks for your efforts providing LTS to debian. Dan - From: Michael Biebl Sent: 06 March 2019 13:18 To: Dan Poltawski; 920...@bugs.debian.org Subject: Re: Bug#920018: Jessie Fix? Am 06.03.19 um 13:17 schrieb Dan Poltawski: > We are seeing this bug on the jessie ( systemd 215-17+deb8u10) and it's > extremely serious, leading to systemd-journald eventually consuming all > memory on the system. Is there a fix for jessie incoming? jessie is oldstable and no longer officially supported directly by Debian. There is a Debian LTS effort. You should probably contact them for a jessie fix. https://wiki.debian.org/LTS Regards, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? The Networking People (NorthWest) Limited. Registered office: c/o Hanleys, Spring Court, Hale, Cheshire, WA14 2UQ. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Re: Serious regression in systemd 215-17+deb8u10
Hi Dan,
> We have discovered that the latest version of systemd uploaded to
> jessie is causing systemd-journald to use an ever increasing amount of
> memory, eventually leading to all available memory being consumed. This
> has been observed on multiple different systems we use which are
> logging quite heavily and have upgraded to the latest systemd package.
> We have some systems which haven't had the new security patches applied
> and are not observing this behaviour - there is a clear correlation
> with the latest version of the systemd package.
So, this is the patch that was applied in systemd 215-17+deb8u10:
Description: journald: do not store the iovec entry for process commandline
on stack
This fixes a crash (CVE-2018-16864) where we
would read the commandline, whose length is under control of the
sending program, and then crash when trying to create a stack
allocation for it.
.
This is a backport of
https://github.com/systemd/systemd/commit/084eeb865ca63887098e0945fb4e93c852b91b0f
Author: Antoine Beaupré
Bug-Debian: https://bugs.debian.org/918841
Origin: Debian
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1653855
Forwarded: not-needed
Last-Update: 2019-01-22
--- systemd-215.orig/src/journal/journald-server.c
+++ systemd-215/src/journal/journald-server.c
@@ -602,7 +602,10 @@ static void dispatch_message_real(
r = get_process_cmdline(ucred->pid, 0, false, &t);
if (r >= 0) {
-x = strappenda("_CMDLINE=", t);
+/* At most _SC_ARG_MAX (2MB usually), which is
+ * too much to put on stack. Let's use a heap
+ * allocation for this one. */
+x = strappend("_CMDLINE=", t);
free(t);
IOVEC_SET_STRING(iovec[n++], x);
}
@@ -716,7 +719,9 @@ static void dispatch_message_real(
r = get_process_comm(object_pid, &t);
if (r >= 0) {
-x = strappenda("OBJECT_COMM=", t);
+/* See above for size limits, only ->cmdline
+ * may be large, so use a heap allocation for it.
*/
+x = strappend("OBJECT_COMM=", t);
free(t);
IOVEC_SET_STRING(iovec[n++], x);
}
I can't immediately see what's up, nor the direct relevance to the
upstream changes listed on #920018, however.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org 🍥 chris-lamb.co.uk
`-
Re: Serious regression in systemd 215-17+deb8u10
Hi,
If I would have to guess, the difference is that strappenda use alloca,
which then use the stack to allocate the string. And the stack is freed
when the function exists, in contrast with allocating memory in heap. maybe
a free(x) is missing from after the IOVEC_SET_STRING
On Wed, Mar 6, 2019 at 8:57 PM Chris Lamb wrote:
> Hi Dan,
>
> > We have discovered that the latest version of systemd uploaded to
> > jessie is causing systemd-journald to use an ever increasing amount of
> > memory, eventually leading to all available memory being consumed. This
> > has been observed on multiple different systems we use which are
> > logging quite heavily and have upgraded to the latest systemd package.
> > We have some systems which haven't had the new security patches applied
> > and are not observing this behaviour - there is a clear correlation
> > with the latest version of the systemd package.
>
> So, this is the patch that was applied in systemd 215-17+deb8u10:
>
> Description: journald: do not store the iovec entry for process
> commandline on stack
> This fixes a crash (CVE-2018-16864) where we
> would read the commandline, whose length is under control of the
> sending program, and then crash when trying to create a stack
> allocation for it.
> .
> This is a backport of
> https://github.com/systemd/systemd/commit/084eeb865ca63887098e0945fb4e93c852b91b0f
> Author: Antoine Beaupré
> Bug-Debian: https://bugs.debian.org/918841
> Origin: Debian
> Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1653855
> Forwarded: not-needed
> Last-Update: 2019-01-22
>
> --- systemd-215.orig/src/journal/journald-server.c
> +++ systemd-215/src/journal/journald-server.c
> @@ -602,7 +602,10 @@ static void dispatch_message_real(
>
> r = get_process_cmdline(ucred->pid, 0, false, &t);
> if (r >= 0) {
> -x = strappenda("_CMDLINE=", t);
> +/* At most _SC_ARG_MAX (2MB usually), which is
> + * too much to put on stack. Let's use a heap
> + * allocation for this one. */
> +x = strappend("_CMDLINE=", t);
> free(t);
> IOVEC_SET_STRING(iovec[n++], x);
> }
> @@ -716,7 +719,9 @@ static void dispatch_message_real(
>
> r = get_process_comm(object_pid, &t);
> if (r >= 0) {
> -x = strappenda("OBJECT_COMM=", t);
> +/* See above for size limits, only ->cmdline
> + * may be large, so use a heap allocation for
> it. */
> +x = strappend("OBJECT_COMM=", t);
> free(t);
> IOVEC_SET_STRING(iovec[n++], x);
> }
>
> I can't immediately see what's up, nor the direct relevance to the
> upstream changes listed on #920018, however.
>
>
> Regards,
>
> --
> ,''`.
> : :' : Chris Lamb
> `. `'` la...@debian.org 🍥 chris-lamb.co.uk
>`-
>
>
Re: Serious regression in systemd 215-17+deb8u10
Chris Lamb wrote:
> Hi Dan,
>> We have discovered that the latest version of systemd uploaded to
>> jessie is causing systemd-journald to use an ever increasing amount of
>> memory, eventually leading to all available memory being consumed. This
>> has been observed on multiple different systems we use which are
>> logging quite heavily and have upgraded to the latest systemd package.
>> We have some systems which haven't had the new security patches applied
>> and are not observing this behaviour - there is a clear correlation
>> with the latest version of the systemd package.
> So, this is the patch that was applied in systemd 215-17+deb8u10:
> Description: journald: do not store the iovec entry for process
> commandline on stack
> This fixes a crash (CVE-2018-16864) where we
> would read the commandline, whose length is under control of the
> sending program, and then crash when trying to create a stack
> allocation for it.
> .
> This is a backport of
> https://github.com/systemd/systemd/commit/084eeb865ca63887098e0945fb4e93c852b91b0f
> Author: Antoine Beaupré
> Bug-Debian: https://bugs.debian.org/918841
> Origin: Debian
> Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1653855
> Forwarded: not-needed
> Last-Update: 2019-01-22
> --- systemd-215.orig/src/journal/journald-server.c
> +++ systemd-215/src/journal/journald-server.c
> @@ -602,7 +602,10 @@ static void dispatch_message_real(
>
> r = get_process_cmdline(ucred->pid, 0, false, &t);
> if (r >= 0) {
> -x = strappenda("_CMDLINE=", t);
> +/* At most _SC_ARG_MAX (2MB usually), which is
> + * too much to put on stack. Let's use a heap
> + * allocation for this one. */
> +x = strappend("_CMDLINE=", t);
> free(t);
> IOVEC_SET_STRING(iovec[n++], x);
> }
> @@ -716,7 +719,9 @@ static void dispatch_message_real(
>
> r = get_process_comm(object_pid, &t);
> if (r >= 0) {
> -x = strappenda("OBJECT_COMM=", t);
> +/* See above for size limits, only ->cmdline
> + * may be large, so use a heap allocation for
> it. */
> +x = strappend("OBJECT_COMM=", t);
> free(t);
> IOVEC_SET_STRING(iovec[n++], x);
> }
> I can't immediately see what's up, nor the direct relevance to the
> upstream changes listed on #920018, however.
The problem with the change for Stretch was that it *always* allocated
2MB on the heap, even when then amount needed was much smaller, until
16384 journal cache entries where filled.
But using my test loop from
https://github.com/systemd/systemd/issues/11502#issuecomment-456407529 I
am not able to reproduce the problem on a Debian Jessie system (AMD64)
with 215-17+deb8u10.
Grüße,
Sven.
--
Sigmentation fault. Core dumped.
Re: Serious regression in systemd 215-17+deb8u10
Thanks for your responses. One of my colleagues has been looking into this trying to get the bottom of it and we do seem to have identified a memory leak which isn't present on stretch. I note the report posted to the list https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924060. Here are what my colleague Alan has documented about his findings, we're going into this without much knowledge of the area, sharing in the hope it helps someone more familiar with the situation: We have run journald under Valgrind, putting system logs through logger - which has identified several memory leaks, for example: ==9577== 1,298,665 bytes in 21,103 blocks are definitely lost in loss record 11 of 11 ==9577== at 0x4C28C20: malloc (vg_replace_malloc.c:296) ==9577== by 0x1272F0: strappend (in /lib/systemd/systemd-journald) ==9577== by 0x112849: dispatch_message_real.lto_priv.175 (in /lib/systemd/systemd-journald) ==9577== by 0x131867: server_dispatch_message (in /lib/systemd/systemd-journald) ==9577== by 0x12F189: process_datagram.part.10.lto_priv.168 (in ==9577== by 0x12355F: source_dispatch (in /lib/systemd/systemd-journald) ==9577== by 0x12402E: sd_event_run (in /lib/systemd/systemd-journald) ==9577== by 0x10DFC3: main (in /lib/systemd/systemd-journald) Note that CVE-2018-16864.patch applies to function "dispatch_message_real" in journald-server.c, which matches the leak found by Valgrind. This patch changes calls to "strappenda" to "strappend". "strappenda" calls "alloca(3)" which allocates memory on the stack, to be freed on function return. "strappend" calls internal function "new()", which ends up calling "malloc_multiply()" and then "malloc(3)" "malloc" normally allocates memory from the heap. There are no corresponding calls to "free(3)" in the patch. Red Hat's initial fix for CVE-2018-16864 introduced the same bug, which has been assigned CVE-2019-3815 See https://access.redhat.com/security/cve/cve-2019-3815 and https://bugzilla.redhat.com/show_bug.cgi?id=146 (currently "Access Denied") Centos has applied the RH patch in 0669-journald-free-cmdline-buffers-owned-by-iovec.patch from http://vault.centos.org/7.6.1810/updates/Source/SPackages/systemd-219-62.el7_6.5.src.rpm although the RH/Centos code is rather different in that they have backported "set_iovec_field_free" from systemd v233 ---------- From: Chris Lamb Sent: 06 March 2019 19:40 To: Dan Poltawski; debian-lts@lists.debian.org Subject: Re: Serious regression in systemd 215-17+deb8u10 Hi Dan, > We have discovered that the latest version of systemd uploaded to > jessie is causing systemd-journald to use an ever increasing amount of > memory, eventually leading to all available memory being consumed. This > has been observed on multiple different systems we use which are > logging quite heavily and have upgraded to the latest systemd package. > We have some systems which haven't had the new security patches applied > and are not observing this behaviour - there is a clear correlation > with the latest version of the systemd package. So, this is the patch that was applied in systemd 215-17+deb8u10: Description: journald: do not store the iovec entry for process commandline on stack This fixes a crash (CVE-2018-16864) where we would read the commandline, whose length is under control of the sending program, and then crash when trying to create a stack allocation for it. . This is a backport of https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsystemd%2Fsystemd%2Fcommit%2F084eeb865ca63887098e0945fb4e93c852b91b0f&data=02%7C01%7Cdan.poltawski%40tnp.net.uk%7Caa9f27e2dcc5426fff0e08d6a26b8bad%7C925e5e137a8344cbb28144e1c700f7e5%7C1%7C0%7C636874980116819527&sdata=eXxpHC6N5HvqMm%2FANrXSTgJ%2B4TrmoXFouYnT8DYHbVg%3D&reserved=0 Author: Antoine Beaupré Bug-Debian: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.debian.org%2F918841&data=02%7C01%7Cdan.poltawski%40tnp.net.uk%7Caa9f27e2dcc5426fff0e08d6a26b8bad%7C925e5e137a8344cbb28144e1c700f7e5%7C1%7C0%7C636874980116819527&sdata=Nc%2FhVL6OtNP9xDC%2FDNd13UBCbGlJQQ%2Fpc5HcTeYegyM%3D&reserved=0 Origin: Debian Bug: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.redhat.com%2Fshow_bug.cgi%3Fid%3D1653855&data=02%7C01%7Cdan.poltawski%40tnp.net.uk%7Caa9f27e2dcc5426fff0e08d6a26b8bad%7C925e5e137a8344cbb28144e1c700f7e5%7C1%7C0%7C636874980116819527&sdata=dxW4DrP2EZkKB2KwDIGp4ujl2j9GP0aGVDFlkco6k%2FY%3D&reserved=0 Forwarded: not-needed Last-Update: 2019-01-22 --- systemd-215.orig/src/journal/journald-server.c +++ systemd-215/src/journal/journald-server.c @@ -602,7 +602,10 @@ static void dispatch_message_real( r = get_proc
Re: Serious regression in systemd 215-17+deb8u10
Am 11.03.19 um 15:51 schrieb Dan Poltawski: > Thanks for your responses. One of my colleagues has been looking into this > trying to get the bottom of it and we do seem to have identified a memory > leak which isn't present on stretch. I note the report posted to the list > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924060. [...] Thank you for sharing your analysis with us. I will prepare a regression update shortly. It appears the confusion stems from the fact that CVE-2018-16864 was already addressed by version 215-17+deb8u9. Thus nobody saw a connection between the memory leak and the recent upload which deals with another issue. Regards, Markus signature.asc Description: OpenPGP digital signature
Re: Serious regression in systemd 215-17+deb8u10
Am 11.03.19 um 19:17 schrieb Markus Koschany: > > Am 11.03.19 um 15:51 schrieb Dan Poltawski: >> Thanks for your responses. One of my colleagues has been looking into this >> trying to get the bottom of it and we do seem to have identified a memory >> leak which isn't present on stretch. I note the report posted to the list >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924060. > > [...] > > Thank you for sharing your analysis with us. I will prepare a regression > update shortly. > It appears the confusion stems from the fact that CVE-2018-16864 was already > addressed > by version 215-17+deb8u9. Thus nobody saw a connection between the memory > leak and the recent > upload which deals with another issue. Thanks, Markus. Also big thanks to the debian-lts team in general for backporting those security fixes for the systemd package in old-stable. Regards, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth?
Re: Serious regression in systemd 215-17+deb8u10
Control: tags -1 pending Hi, Am 11.03.19 um 23:16 schrieb Michael Biebl: [...] > > Thanks, Markus. > > Also big thanks to the debian-lts team in general for backporting those > security fixes for the systemd package in old-stable. I could reproduce the memory leak with valgrind following the steps provided by Dan. Indeed this issue is also known as CVE-2019-3815. I am going to upload a new revision of systemd in Jessie now. Thank you all for your feedback. Regards, Markus signature.asc Description: OpenPGP digital signature
