On 2011-11-28 Stefan Westerfeld wrote: > Now the problem is that for those files I cannot predict the size. > Often they will be quite small, but they also could be 100 MB in size > or more. So I use xz -9 to get the best compression. > > The problem is now that xz takes a lot of time to start: > > stefan@ubuntu:/tmp$ time echo "foo" | xz -9 >/dev/null > > real 0m0.155s > user 0m0.052s > sys 0m0.096s
The match finder hash table has to be initialized. It cannot be avoided. The bigger the dictionary, the bigger the hash table. It's about 64 MiB when using 64 MiB dictionary (xz -9). With 8 MiB dictionary (xz -6) it's about 16 MiB. So at a lower setting the initialization is faster. xz allocates much more memory for other things. Most of that memory isn't initialized beforehand. Uninitialized memory doesn't cause a significant speed penalty because many kernels don't physically allocate large allocations before the memory will actually be used. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
