On 2022-09-28 Jia Tan wrote: > On 2022-09-27 Sebastian Andrzej Siewior wrote: > > Okay, so that is what you are tracking. I remember that there was a > > stall in the decoding but I don't remember how it played out. > > > > I do remember that I had something for memory allocation/ limit but > > I don't remember if we settled on something or if discussion is > > needed. Also how many decoding threads make sense, etc. > > We ended up changing xz to use (total_ram / 4) as the default "soft > limit". If the soft limit is reached, xz will decode single threaded. > The "hard limit" shares the same environment variable and xz option > (--memlimit-decompress).
There is also the 1400 MiB cap for 32-bit executables. The memory limiting in threaded decompression (two separate limits in parallel) is one thing where feedback would be important as after the liblzma API, ABI and xz tool syntax are in a stable release, backward compatibility has to be maintained. Another thing needing feedback is the new behavior of -T0 when no memlimit has been specified. Now it has a default soft limit. I hope it is an improvement but quite possibly it could be improved. Your suggestion to use MemAvailable on Linux is one thing that could be included if people think it is a good way to go as a Linux-specific behavior (having more benefits than downsides). These are documented on the xz man page. I hope it is clear enough. It feels a bit complicated, which is a bad sign but on the other hand I feel the underlying problem isn't as trivial as it seems on the surface. So far Jia Tan and I have received no feedback about these things at all. I would prefer to hear the complaints before 5.4.0 is out. :-) > > This reminds me that I once posted a patch to use openssl for the > > sha256. > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00429.html > > > > Some distro is using sha256 instead crc64 by default, I don't > > remember which one… Not that I care personally ;) > > I am unsure if we will have time to include your sha256 patch, but if > we finish all the tasks with extra time it may be considered. There's more to this than available time. 5.1.2alpha added support for using SHA-256 from the OS base libraries (not OpenSSL) but starting with 5.2.3 it is disabled by default. Some OS libs use (or used to use) the same symbol names for SHA-256 functions as OpenSSL while having incompatible ABI. This lead to weird problems when an application needed both liblzma and OpenSSL as liblzma ended up calling OpenSSL functions. Plus, some of the OS-specific implementations were slower than the C code in liblzma (OpenSSL would be faster). OpenSSL's license has compatibility questions with GNU GPL. If I remember correctly, some distributions consider OpenSSL to be part of the core operating system and thus avoid the compatibility problem with the GPL. I'm not up to date how distros handle it in 2022 but perhaps it should be taken into account so that apps depending on liblzma won't get legally unacceptable OpenSSL linkage. So if OpenSSL support is added it likely should be disabled by default in configure.ac. > > > This is everything currently planned. Translations need to be updated too once the strings and man pages are close to final. A development release needs to be sent to the Translation Project at some point. If people want to translate the man pages too, they will need quite a bit of time. -- Lasse Collin
