Lasse, On Sat, Dec 26, 2020 at 05:04:02PM +0200, Lasse Collin wrote: > On 2020-12-26 Vitaly Chikunov wrote: > > This wasn't working, because `memlimit_compress` initialized with > > zero, thus memory limit is never lowered for 32-bit address space, > > causing `Cannot allocate memory' error (in `lzma_outq_init()'). For > > example, when `-T0' is used on 32 CPUs with compression level higher > > than `-6'. > > Zero means disabling the limiter. That is the default. The patch would > change this and enable a memory usage limit by default on 32-bit > platforms. > > If you want to enable a limiter by default, you may use the environment > variables XZ_DEFAULTS or XZ_OPT (see the man page). > > The whole memory usage limiting feature is a bit controversial and I'm > hesitant to enable anything by default (although with threaded > decompression there may need to be some default *soft* limit). Your > suggestion is reasonable but someone else may prefer that xz gives a > hard error if too high settings are specified instead of xz trying to > outsmart the user and change the settings automatically. Perhaps that > sounds silly but I don't have energy for endless arguments where both > sides are kind of right but both wishes cannot be fulfilled at the > same time. > > I cannot make everyone happy.
Wow, that's philosophical! I think, we should solve this fundamental problem first. -- Even if we cannot satisfy everybody, better than satisfying just one party and make other unhappy, we can give users choice. If that's approach is accepted we can rework patch to make it better. Then, I think, better approach is 'sane defaults', thus more use cases are working correctly out of the box. (Most users don't need to test memory subsystem they need compression from xz.) Users who want (by hard to imagine unknown reason) xz to fail on 32-bit arches would use `--no-adjust` -- we could (rework the patch to) disable memory auto-fitting logic if this option is used. Does it sound good? Enabling (robust behavior) by default is better than not, because -- you do not shift burden to users: 1) unexpected consequences (of xz not working in some corner cases, somewhere), 2) then, user would have hard time solving the issue (man, googling, reading sources). -- By what combination of options of xz to solve this (if she is not expert in xz usage hacks). For example, percentage memory limit on 32-bit systems is calculated against whole memory and not against 'physical' 4MiB limit -- user should somehow find this, probably by trial and error, wasting her time. So, to make `xz -T0` "just work" on all possible systems, single portable approach is just disabling `-T0` and wasting opportinity to speed up compression real time. Second option is to make correct xz option generator depending on system configuration -- she should understand virtual memory and how xz works, and still may miss something. By this, I think its always better that program works by default. Your suggestion (in previous mail) to use `--memlimit-compress=100%` seems to offset this patch. But, still, need rare expert to know about this hack. Also, it would be easier for users (I'm also suggested be colleague) if `--memlimit-compress=100%` is setting enabled by default, and if someone wants to disable limiter she would use `--memlimit-compress=0`. I reason like this: Setting [non-zero value to] `--memlimit-compress=` _increases_ use cases by avoiding memory errors, in comparison to not setting it [or setting it to 0]. So it should be enabled by default. Thanks, > > -- > Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
