Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Adrian Bunk]

On Sun, Dec 04, 2016 at 05:20:09PM +0100, Guillem Jover wrote:
> On Sun, 2016-12-04 at 16:07:16 +0200, Adrian Bunk wrote:
>...
> > I don't know how much RAM the amd64/i386 buildds have,
> > but I'd guess more than 4 GB...
> > 
> > A hard upper limit somewhere around 1.5 GB on all 32 bit architectures
> > (or on all architectures, if that's easier to implement) is required.
> 
> The patch now clamps the physical memory to INTPTR_MAX. So that we both
> do not exceed the physical memory available nor the addressable limit.
> 
> But I should indeed probably also subtract some space for the rest of
> the userland that might be running. I'll do that later today.
>...

It is a bit more complicated.

Clamping to something like 80% of INTPTR_MAX might be good enough
for practical purposes, but technically this limit is not related
to INTPTR_MAX.

Addressable are actually 4 GB.

The problem is that these 4 GB are split between user memory and
kernel lowmem.

On several architectures (e.g. i386, ARM) it is actually a configure 
option of the kernel whether you want 1 GB, 2 GB or 3 GB of user memory
(with accordingly less kernel lowmem).

< 2 GB user memory is unlikely to be a configuration you will see
in practice (especially not on the buildds), so staying some margin
below 2 GB on 32 bit (e.g. 80% of INTPTR_MAX) should work fine in
practice.

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Guillem Jover]

On Sun, 2016-12-04 at 16:07:16 +0200, Adrian Bunk wrote:
> On Sun, Dec 04, 2016 at 05:25:49AM +0100, Guillem Jover wrote:
> > On Fri, 2016-12-02 at 10:31:58 +0100, Guillem Jover wrote:
> > > Right, this was reported the other day on IRC by Mattia Rizzolo. The
> > > combination of -Sextreme -z9 and parallel xz makes this use more than
> > > the available address space. I'll change the code to limit based on
> > > memory available. Although as was mentioned on a thread on d-d, those
> > > settings are pretty unfriendly IMO, even more for memory constrained
> > > arches, but oh well. dpkg should never fail to operate on those
> > > conditions.
> > 
> > I've got the attached patch now, but I've been unable to test that
> > specific incarnation as I don't have 32-bit machine with many cores.
> > And neither are the i386 nor armhf porter boxes. I've just verified
> > that it does what it is intended by hardcoding the number of threads
> > to 32 and setting the physical memory limit to 2 GiB. And it reduced
> > the threads down to 12 when building one of those font packages.

> Your patch won't solve it - the problem is not physical memory.
> 
> The problem is that on 32 bit you cannot use more than 2-3 GB
> (depending on the architecture) in one process.

Right, I'm aware of the distinction, but I guess I was wrongly
assuming the lzma_physmem() function would return (userspace)
addressable physical memory. Checking it now, it seems like not,
as it uses stuff like sysconf(_SC_PHYS_PAGES).

> #845757 was on a mipsel buildd with 4 cores [1] and 8 GB RAM.

Ah, thanks, I'm testing as I write this a new patch I've prepared on
eller.d.o (mipsel) which seems to have the right conditions.

> I don't know how much RAM the amd64/i386 buildds have,
> but I'd guess more than 4 GB...
> 
> A hard upper limit somewhere around 1.5 GB on all 32 bit architectures
> (or on all architectures, if that's easier to implement) is required.

The patch now clamps the physical memory to INTPTR_MAX. So that we both
do not exceed the physical memory available nor the addressable limit.

But I should indeed probably also subtract some space for the rest of
the userland that might be running. I'll do that later today.

> (I'd also suggest a MBF for all the packages that mess with the 
>  compressor for no good reason - just like the ones using bzip2
>  this only causes troubles.)

This has been brought up on d-d some time ago. And many of the
packages affected do not even really benefit from such extreme
settings as the dictionary size is bigger than the actual data
to compress. There's a bug in lintian to report an error on this.

Thanks,
Guillem
From f48c09879e6ba91110a2918a3caea5985ef57a3f Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Sun, 4 Dec 2016 02:35:27 +0100
Subject: [PATCH] libdpkg: Decrease xz encoder threads to not exceed memory
 limits

---
 lib/dpkg/compress.c | 27 ++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/lib/dpkg/compress.c b/lib/dpkg/compress.c
index 2eda658fa..a6dd522ed 100644
--- a/lib/dpkg/compress.c
+++ b/lib/dpkg/compress.c
@@ -529,9 +529,9 @@ filter_xz_init(struct io_lzma *io, lzma_stream *s)
 	uint32_t preset;
 	lzma_check check = LZMA_CHECK_CRC64;
 #ifdef HAVE_LZMA_MT
+	uint64_t mt_memlimit;
 	lzma_mt mt_options = {
 		.flags = 0,
-		.threads = sysconf(_SC_NPROCESSORS_ONLN),
 		.block_size = 0,
 		.timeout = 0,
 		.filters = NULL,
@@ -548,6 +548,31 @@ filter_xz_init(struct io_lzma *io, lzma_stream *s)
 
 #ifdef HAVE_LZMA_MT
 	mt_options.preset = preset;
+
+	/* Initialize the multi-threaded memory limit to half the physical
+	 * RAM, or to 128 MiB if we cannot infer the number. */
+	mt_memlimit = lzma_physmem() / 2;
+	if (mt_memlimit == 0)
+		mt_memlimit = 128 * 1024 * 1024;
+	/* Clamp the multi-threaded memory limit to half the addressable
+	 * memory on this architecture. */
+	if (mt_memlimit > INTPTR_MAX)
+		mt_memlimit = INTPTR_MAX;
+
+	mt_options.threads = lzma_cputhreads();
+	if (mt_options.threads == 0)
+		mt_options.threads = 1;
+
+	/* Check that we have enough RAM to use the multi-threaded encoder,
+	 * and decrease them up to single-threaded to reduce memory usage. */
+	for (; mt_options.threads > 1; mt_options.threads--) {
+		uint64_t mt_memusage;
+
+		mt_memusage = lzma_stream_encoder_mt_memusage(_options);
+		if (mt_memusage < mt_memlimit)
+			break;
+	}
+
 	ret = lzma_stream_encoder_mt(s, _options);
 #else
 	ret = lzma_easy_encoder(s, preset, check);
-- 
2.11.0.rc1.160.g51e66c2

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Adrian Bunk]

On Sun, Dec 04, 2016 at 05:25:49AM +0100, Guillem Jover wrote:
> Hi!
> 
> On Fri, 2016-12-02 at 10:31:58 +0100, Guillem Jover wrote:
> > Right, this was reported the other day on IRC by Mattia Rizzolo. The
> > combination of -Sextreme -z9 and parallel xz makes this use more than
> > the available address space. I'll change the code to limit based on
> > memory available. Although as was mentioned on a thread on d-d, those
> > settings are pretty unfriendly IMO, even more for memory constrained
> > arches, but oh well. dpkg should never fail to operate on those
> > conditions.
> 
> I've got the attached patch now, but I've been unable to test that
> specific incarnation as I don't have 32-bit machine with many cores.
> And neither are the i386 nor armhf porter boxes. I've just verified
> that it does what it is intended by hardcoding the number of threads
> to 32 and setting the physical memory limit to 2 GiB. And it reduced
> the threads down to 12 when building one of those font packages.
>...

Your patch won't solve it - the problem is not physical memory.

The problem is that on 32 bit you cannot use more than 2-3 GB
(depending on the architecture) in one process.

#845757 was on a mipsel buildd with 4 cores [1] and 8 GB RAM.

I don't know how much RAM the amd64/i386 buildds have,
but I'd guess more than 4 GB...

A hard upper limit somewhere around 1.5 GB on all 32 bit architectures
(or on all architectures, if that's easier to implement) is required.

(I'd also suggest a MBF for all the packages that mess with the 
 compressor for no good reason - just like the ones using bzip2
 this only causes troubles.)

> Thanks,
> Guillem

cu
Adrian
 
[1] 4x 674 MiB with -9 > 2 GB limit on 32bit mips/mipsel

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Mattia Rizzolo]

On Sun, Dec 04, 2016 at 10:01:27AM +, Holger Levsen wrote:
> > If someone could test this on such 32-bit machine, that would be
> > appreciated.
> 
> Maybe Mattia can test that? (and what exactly? Apply the patch here,
> sure and then build which package? (for the unlikely case that I'll find
> time soon…)

I can surely find the time to test it.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Holger Levsen]

On Sun, Dec 04, 2016 at 05:25:49AM +0100, Guillem Jover wrote:
> On Fri, 2016-12-02 at 10:31:58 +0100, Guillem Jover wrote:
> > Right, this was reported the other day on IRC by Mattia Rizzolo. The
> > combination of -Sextreme -z9 and parallel xz makes this use more than
> > the available address space. I'll change the code to limit based on
> > memory available. Although as was mentioned on a thread on d-d, those
> > settings are pretty unfriendly IMO, even more for memory constrained
> > arches, but oh well. dpkg should never fail to operate on those
> > conditions.
> 
> I've got the attached patch now, but I've been unable to test that
> specific incarnation as I don't have 32-bit machine with many cores.
> And neither are the i386 nor armhf porter boxes. I've just verified
> that it does what it is intended by hardcoding the number of threads
> to 32 and setting the physical memory limit to 2 GiB. And it reduced
> the threads down to 12 when building one of those font packages.
> 
> If someone could test this on such 32-bit machine, that would be
> appreciated.

I fear I dont really have the time to test this, but the i386 nodes used
for reproducible builds testing have 10 cores and 36gb ram, plus we have
several quad and octo core armhf systems (with 2-4gb ram).

Maybe Mattia can test that? (and what exactly? Apply the patch here,
sure and then build which package? (for the unlikely case that I'll find
time soon…)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Guillem Jover]

Hi!

On Fri, 2016-12-02 at 10:31:58 +0100, Guillem Jover wrote:
> Right, this was reported the other day on IRC by Mattia Rizzolo. The
> combination of -Sextreme -z9 and parallel xz makes this use more than
> the available address space. I'll change the code to limit based on
> memory available. Although as was mentioned on a thread on d-d, those
> settings are pretty unfriendly IMO, even more for memory constrained
> arches, but oh well. dpkg should never fail to operate on those
> conditions.

I've got the attached patch now, but I've been unable to test that
specific incarnation as I don't have 32-bit machine with many cores.
And neither are the i386 nor armhf porter boxes. I've just verified
that it does what it is intended by hardcoding the number of threads
to 32 and setting the physical memory limit to 2 GiB. And it reduced
the threads down to 12 when building one of those font packages.

If someone could test this on such 32-bit machine, that would be
appreciated.

Thanks,
Guillem
From 981fe7b1cc828e9d446bc07298264e421a970ac9 Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Sun, 4 Dec 2016 02:35:27 +0100
Subject: [PATCH] libdpkg: Decrease xz encoder threads to not exceed memory
 limits

---
 lib/dpkg/compress.c | 23 ++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/lib/dpkg/compress.c b/lib/dpkg/compress.c
index 2eda658fa..05e056b22 100644
--- a/lib/dpkg/compress.c
+++ b/lib/dpkg/compress.c
@@ -529,9 +529,9 @@ filter_xz_init(struct io_lzma *io, lzma_stream *s)
 	uint32_t preset;
 	lzma_check check = LZMA_CHECK_CRC64;
 #ifdef HAVE_LZMA_MT
+	uint64_t mt_memlimit;
 	lzma_mt mt_options = {
 		.flags = 0,
-		.threads = sysconf(_SC_NPROCESSORS_ONLN),
 		.block_size = 0,
 		.timeout = 0,
 		.filters = NULL,
@@ -548,6 +548,27 @@ filter_xz_init(struct io_lzma *io, lzma_stream *s)
 
 #ifdef HAVE_LZMA_MT
 	mt_options.preset = preset;
+
+	/* Set the multi-threaded memory limit to half the physical RAM,
+	 * or to 128 MiB if we cannot infer the number. */
+	mt_memlimit = lzma_physmem() / 2;
+	if (mt_memlimit == 0)
+		mt_memlimit = 128 * 1024 * 1024;
+
+	mt_options.threads = lzma_cputhreads();
+	if (mt_options.threads == 0)
+		mt_options.threads = 1;
+
+	/* Check that we have enough RAM to use the multi-threaded encoder,
+	 * and decrease them up to single-threaded to reduce memory usage. */
+	for (; mt_options.threads > 1; mt_options.threads--) {
+		uint64_t mt_memusage;
+
+		mt_memusage = lzma_stream_encoder_mt_memusage(_options);
+		if (mt_memusage < mt_memlimit)
+			break;
+	}
+
 	ret = lzma_stream_encoder_mt(s, _options);
 #else
 	ret = lzma_easy_encoder(s, preset, check);
-- 
2.11.0.rc1.160.g51e66c2

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Guillem Jover]

Hi!

On Fri, 2016-12-02 at 09:58:44 +0100, Andreas Beckmann wrote:
> Package: dpkg
> Version: 1.18.15
> Severity: serious

> I just tried to rebuild some packages from non-free. fonts-larabie
> builds on amd64 but fails on i386 and armhf. This still worked a month
> ago. (building on a amd64 machine in pbuilder chroots, armhf using qemu)
> 
>debian/rules override_dh_builddeb
> make[1]: Entering directory '/build/fonts-larabie-20011216'
> dh_builddeb -- -Zxz -Sextreme -z9
> dpkg-deb: building package 'fonts-larabie-straight' in 
> '../fonts-larabie-straight_20011216-5_all.deb'.
> dpkg-deb: building package 'fonts-larabie-deco' in 
> '../fonts-larabie-deco_20011216-5_all.deb'.
> dpkg-deb: building package 'fonts-larabie-uncommon' in 
> '../fonts-larabie-uncommon_20011216-5_all.deb'.
> dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate 
> memory
> dpkg-deb: error: subprocess  from tar -cf returned error exit 
> status 2
> dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate 
> memory
> dh_builddeb: dpkg-deb -Zxz -Sextreme -z9 --build 
> debian/fonts-larabie-straight .. returned exit code 2
> dpkg-deb: error: subprocess  from tar -cf returned error exit 
> status 2
> dh_builddeb: dpkg-deb -Zxz -Sextreme -z9 --build debian/fonts-larabie-deco .. 
> returned exit code 2
> dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate 
> memory
> dpkg-deb: error: subprocess  from tar -cf returned error exit 
> status 2
> dh_builddeb: dpkg-deb -Zxz -Sextreme -z9 --build 
> debian/fonts-larabie-uncommon .. returned exit code 2
> debian/rules:16: recipe for target 'override_dh_builddeb' failed
> make[1]: *** [override_dh_builddeb] Error 1
> make[1]: Leaving directory '/build/fonts-larabie-20011216'

Right, this was reported the other day on IRC by Mattia Rizzolo. The
combination of -Sextreme -z9 and parallel xz makes this use more than
the available address space. I'll change the code to limit based on
memory available. Although as was mentioned on a thread on d-d, those
settings are pretty unfriendly IMO, even more for memory constrained
arches, but oh well. dpkg should never fail to operate on those
conditions.

Thanks,
Guillem

Bug#846564: dpkg-deb: 'compressing tar member: lzma error: Cannot allocate memory' on 32-bit architectures on dpkg-deb -Zxz -Sextreme -z9

[Andreas Beckmann]

Package: dpkg
Version: 1.18.15
Severity: serious

Hi,

I just tried to rebuild some packages from non-free. fonts-larabie
builds on amd64 but fails on i386 and armhf. This still worked a month
ago. (building on a amd64 machine in pbuilder chroots, armhf using qemu)

   debian/rules override_dh_builddeb
make[1]: Entering directory '/build/fonts-larabie-20011216'
dh_builddeb -- -Zxz -Sextreme -z9
dpkg-deb: building package 'fonts-larabie-straight' in 
'../fonts-larabie-straight_20011216-5_all.deb'.
dpkg-deb: building package 'fonts-larabie-deco' in 
'../fonts-larabie-deco_20011216-5_all.deb'.
dpkg-deb: building package 'fonts-larabie-uncommon' in 
'../fonts-larabie-uncommon_20011216-5_all.deb'.
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate 
memory
dpkg-deb: error: subprocess  from tar -cf returned error exit status 2
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate 
memory
dh_builddeb: dpkg-deb -Zxz -Sextreme -z9 --build debian/fonts-larabie-straight 
.. returned exit code 2
dpkg-deb: error: subprocess  from tar -cf returned error exit status 2
dh_builddeb: dpkg-deb -Zxz -Sextreme -z9 --build debian/fonts-larabie-deco .. 
returned exit code 2
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate 
memory
dpkg-deb: error: subprocess  from tar -cf returned error exit status 2
dh_builddeb: dpkg-deb -Zxz -Sextreme -z9 --build debian/fonts-larabie-uncommon 
.. returned exit code 2
debian/rules:16: recipe for target 'override_dh_builddeb' failed
make[1]: *** [override_dh_builddeb] Error 1
make[1]: Leaving directory '/build/fonts-larabie-20011216'


Andreas