Hi!
On Fri, 2016-12-02 at 10:31:58 +0100, Guillem Jover wrote:
> Right, this was reported the other day on IRC by Mattia Rizzolo. The
> combination of -Sextreme -z9 and parallel xz makes this use more than
> the available address space. I'll change the code to limit based on
> memory available. Although as was mentioned on a thread on d-d, those
> settings are pretty unfriendly IMO, even more for memory constrained
> arches, but oh well. dpkg should never fail to operate on those
> conditions.
I've got the attached patch now, but I've been unable to test that
specific incarnation as I don't have 32-bit machine with many cores.
And neither are the i386 nor armhf porter boxes. I've just verified
that it does what it is intended by hardcoding the number of threads
to 32 and setting the physical memory limit to 2 GiB. And it reduced
the threads down to 12 when building one of those font packages.
If someone could test this on such 32-bit machine, that would be
appreciated.
Thanks,
Guillem
From 981fe7b1cc828e9d446bc07298264e421a970ac9 Mon Sep 17 00:00:00 2001
From: Guillem Jover <guil...@debian.org>
Date: Sun, 4 Dec 2016 02:35:27 +0100
Subject: [PATCH] libdpkg: Decrease xz encoder threads to not exceed memory
limits
---
lib/dpkg/compress.c | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/lib/dpkg/compress.c b/lib/dpkg/compress.c
index 2eda658fa..05e056b22 100644
--- a/lib/dpkg/compress.c
+++ b/lib/dpkg/compress.c
@@ -529,9 +529,9 @@ filter_xz_init(struct io_lzma *io, lzma_stream *s)
uint32_t preset;
lzma_check check = LZMA_CHECK_CRC64;
#ifdef HAVE_LZMA_MT
+ uint64_t mt_memlimit;
lzma_mt mt_options = {
.flags = 0,
- .threads = sysconf(_SC_NPROCESSORS_ONLN),
.block_size = 0,
.timeout = 0,
.filters = NULL,
@@ -548,6 +548,27 @@ filter_xz_init(struct io_lzma *io, lzma_stream *s)
#ifdef HAVE_LZMA_MT
mt_options.preset = preset;
+
+ /* Set the multi-threaded memory limit to half the physical RAM,
+ * or to 128 MiB if we cannot infer the number. */
+ mt_memlimit = lzma_physmem() / 2;
+ if (mt_memlimit == 0)
+ mt_memlimit = 128 * 1024 * 1024;
+
+ mt_options.threads = lzma_cputhreads();
+ if (mt_options.threads == 0)
+ mt_options.threads = 1;
+
+ /* Check that we have enough RAM to use the multi-threaded encoder,
+ * and decrease them up to single-threaded to reduce memory usage. */
+ for (; mt_options.threads > 1; mt_options.threads--) {
+ uint64_t mt_memusage;
+
+ mt_memusage = lzma_stream_encoder_mt_memusage(&mt_options);
+ if (mt_memusage < mt_memlimit)
+ break;
+ }
+
ret = lzma_stream_encoder_mt(s, &mt_options);
#else
ret = lzma_easy_encoder(s, preset, check);
--
2.11.0.rc1.160.g51e66c2
