Package: lbzip2 Version: 0.23-1 Severity: important lbzip2 uses excessive amounts of memory (and is killed) when decompressing highly compressible data and the writer (aka: "muxer") is slow.
I recently tried to decompress some hard drive images, which have large sections of zero filled area, unfortunately due to runaway memory usage these files were uncompressable with lbzip2. What I think is happening, the splitter pulls some blocks of highly compressed input into ram, a worker gets one and enters scan mode, which causes it to very rapidly find and queue up many of the small compressed blocks for decompression. Nothing in the code directly throttles output block creation between decompressor and muxer, so if the muxer can't write out blocks quickly enough to keep up the process size explodes. The throttling between splitter and muxer does not help here because the just one splitter is around 16G of output, even if the splitter loads only one block I won't have enough ram to decompress into ram that single input block. The simple test case below, use 100G of compressed zeros. This file has a little more than 6 blocks of input from splitter's perspective. On a machine with 6G of ram and 4 processors, it takes about 10 seconds to be killed, at that time it had used 20G of virtual memory and 6G of resident memory. The kill is fast for me because I do not have swap enabled on this machine. If swap even a small (2G) swap was enabled the system will trash various things to disk for several minutes before the OOM-killer triggers... so if you are on linux and have swap enabled, I recommend monitoring top and hitting ^C on lbzip2 when you run the test case. The simple test case: === # generate input file for problem report dd if=/dev/zero bs=1024k count=$[1024*100] | ./lbzip2 -c9 > zero.bz2 102400+0 records in 102400+0 records out 107374182400 bytes (107 GB) copied, 648.662 s, 166 MB/s # check size of file ls -l zero.bz2 -rw-r--r-- 1 user user 5726783 2011-10-20 04:44 zero.bz2 # reproduce problem, the kill -9 /usr/bin/time ./lbzip2 -cd zero.bz2 | sleep 30 Command terminated by signal 9 22.40user 5.72system 0:10.48elapsed 268%CPU (0avgtext+0avgdata 24060336maxresident)k 12312inputs+0outputs (11major+1508100minor)pagefaults 0swaps # confirm OOM kill in dmesg dmesg | grep lbzip2 | tail -2 [622439.076387] Out of memory: Kill process 10299 (lbzip2) score 739 or sacrifice child [622439.076391] Killed process 10299 (lbzip2) total-vm:20247368kB, anon-rss:6015048kB, file-rss:4kB === With this example input the muxer writing to /dev/null can't keep up with just two workers... this is killed: lbzip2 -cd -n 2 zero.bz2 > /dev/null Even running in single worker mode is killed: lbzip2 -cd -n 1 zero.bz2 | sleep 60 -- System Information: Debian Release: squeeze/sid APT prefers natty-updates APT policy: (500, 'natty-updates'), (500, 'natty-security'), (500, 'natty') Architecture: amd64 (x86_64) Kernel: Linux 2.6.38-11-generic (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages lbzip2 depends on: ii libbz2-1.0 1.0.5-6ubuntu1 high-quality block-sorting file co ii libc6 2.13-0ubuntu13 Embedded GNU C Library: Shared lib lbzip2 recommends no packages. lbzip2 suggests no packages. -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
