Hello!
On 2020-12-24 Vitaly Chikunov wrote:
> Add guard call to allocate the memory before calling get_thread to
> prevent memory failures (LZMA_MEM_ERROR).
>
> This would make simple `xz -T0' more robust on 32-bit architectures.
>
> Rationale: simple `-T0' is hard to use portably in scripts on
> different platforms, because there is always different amount of RAM
> and CPU, causing unexpected "xz: Cannot allocate memory" errors.
Yes, -T0 is a bit broken as on systems with many cores it may attempt
to use ridiculous amount of resources. Enabling a memory usage limit
can help, and starting with xz 5.2.5 there is a small hack that makes
memory usage limiting work better with 32-bit xz binaries when running
under a 64-bit kernel. From the NEWS file:
- Added a special case for 32-bit xz: If --memlimit-compress is
used to specify a limit that exceeds 4020 MiB, the limit will
be set to 4020 MiB. The values "0" and "max" aren't affected
by this and neither is decompression. This hack can be
helpful when a 32-bit xz has access to 4 GiB address space
but the specified memlimit exceeds 4 GiB. This can happen
e.g. with some scripts.
Thus one can use "xz -T0 --memlimit-compress=100%" or 90% or whatever
number one finds good. If that percentage results in a value over 4020
MiB with a 32-bit xz, the limit is reduced to 4020 MiB. This should
make things work quite often.
Have you tried something like the above or does it have some other
problem?
> + // Test memory availability.
> + void *mem = lzma_alloc(coder->block_size * 6, allocator);
> + if (!mem) {
> + coder->threads_max = coder->threads_initialized;
> + return LZMA_OK;
> + } else
> + lzma_free(mem, allocator);
> +
This is a too ugly hack. It makes assumptions that aren't always true.
A better way would be to add an option to make the encoder tolerate
allocation failures and be able to continue with fewer threads.
However, I haven't heard complaints about this outside xz -T0 context
and I think the existing hack isn't horrible.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
