Your message dated Mon, 20 Aug 2012 10:17:37 +0000
with message-id <[email protected]>
and subject line Bug#645999: fixed in lbzip2 2.2-1
has caused the Debian Bug report #645999,
regarding lbzip2: uses excessive amounts of memory when decompressing highly
compressible data
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
645999: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645999
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: lbzip2
Version: 0.23-1
Severity: important
lbzip2 uses excessive amounts of memory (and is killed) when decompressing
highly compressible data and the writer (aka: "muxer") is slow.
I recently tried to decompress some hard drive images, which have large
sections of zero filled area, unfortunately due to runaway memory usage
these files were uncompressable with lbzip2.
What I think is happening, the splitter pulls some blocks of highly
compressed input into ram, a worker gets one and enters scan mode,
which causes it to very rapidly find and queue up many of the small
compressed blocks for decompression. Nothing in the code directly
throttles output block creation between decompressor and muxer,
so if the muxer can't write out blocks quickly enough to keep up
the process size explodes.
The throttling between splitter and muxer does not help here because
the just one splitter is around 16G of output, even if the splitter
loads only one block I won't have enough ram to decompress into
ram that single input block.
The simple test case below, use 100G of compressed zeros. This file
has a little more than 6 blocks of input from splitter's perspective.
On a machine with 6G of ram and 4 processors, it takes about 10 seconds
to be killed, at that time it had used 20G of virtual memory and 6G of
resident memory.
The kill is fast for me because I do not have swap enabled on this machine.
If swap even a small (2G) swap was enabled the system will trash various
things to disk for several minutes before the OOM-killer triggers... so
if you are on linux and have swap enabled, I recommend monitoring top
and hitting ^C on lbzip2 when you run the test case.
The simple test case:
===
# generate input file for problem report
dd if=/dev/zero bs=1024k count=$[1024*100] | ./lbzip2 -c9 > zero.bz2
102400+0 records in
102400+0 records out
107374182400 bytes (107 GB) copied, 648.662 s, 166 MB/s
# check size of file
ls -l zero.bz2
-rw-r--r-- 1 user user 5726783 2011-10-20 04:44 zero.bz2
# reproduce problem, the kill -9
/usr/bin/time ./lbzip2 -cd zero.bz2 | sleep 30
Command terminated by signal 9
22.40user 5.72system 0:10.48elapsed 268%CPU (0avgtext+0avgdata
24060336maxresident)k
12312inputs+0outputs (11major+1508100minor)pagefaults 0swaps
# confirm OOM kill in dmesg
dmesg | grep lbzip2 | tail -2
[622439.076387] Out of memory: Kill process 10299 (lbzip2) score 739 or
sacrifice child
[622439.076391] Killed process 10299 (lbzip2) total-vm:20247368kB,
anon-rss:6015048kB, file-rss:4kB
===
With this example input the muxer writing to /dev/null can't keep
up with just two workers... this is killed:
lbzip2 -cd -n 2 zero.bz2 > /dev/null
Even running in single worker mode is killed:
lbzip2 -cd -n 1 zero.bz2 | sleep 60
-- System Information:
Debian Release: squeeze/sid
APT prefers natty-updates
APT policy: (500, 'natty-updates'), (500, 'natty-security'), (500, 'natty')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.38-11-generic (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages lbzip2 depends on:
ii libbz2-1.0 1.0.5-6ubuntu1 high-quality block-sorting file co
ii libc6 2.13-0ubuntu13 Embedded GNU C Library: Shared lib
lbzip2 recommends no packages.
lbzip2 suggests no packages.
-- no debconf information
--- End Message ---
--- Begin Message ---
Source: lbzip2
Source-Version: 2.2-1
We believe that the bug you reported is fixed in the latest version of
lbzip2, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Mikolaj Izdebski <[email protected]> (supplier of updated lbzip2 package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Format: 1.8
Date: Mon, 20 Aug 2012 07:29:04 +0200
Source: lbzip2
Binary: lbzip2
Architecture: source amd64
Version: 2.2-1
Distribution: unstable
Urgency: low
Maintainer: Mikolaj Izdebski <[email protected]>
Changed-By: Mikolaj Izdebski <[email protected]>
Description:
lbzip2 - fast, multi-threaded bzip2 utility
Closes: 582476 645999 673378
Changes:
lbzip2 (2.2-1) unstable; urgency=low
.
* New upstream release:
- limited memory allocation, closes: #645999,
- improved bzip2 compatibility, closes: #582476,
- fixed several other minor bugs, closes: #673378.
* debian/control:
- drop version requirements on autotools,
- bump Standards-Version to 3.9.3.
* debian/copyright:
- remove comment about maintainers involved in creation of the package,
- update to reflect new upstream version.
* debian/rules: execute Bourne shell scripts with sh instead of perl.
* debian/compat: Bump to 9.
* debian/watch: Use github.com directly instead of githubredir.
Checksums-Sha1:
b72d0a1cdb85a3a080a637459a44577b13f25a92 1169 lbzip2_2.2-1.dsc
6493d360c52d695d4749d836b1e5ec98f43dfade 134040 lbzip2_2.2.orig.tar.gz
25ea3b2f1cd628e953140890bad6de3288d3f1f2 5352 lbzip2_2.2-1.debian.tar.bz2
6d5cc52d4207063b64da751324929c4e9127f254 79960 lbzip2_2.2-1_amd64.deb
Checksums-Sha256:
5f608fc31e0ced441fde050cce46a692091a7502ed24abef98f07fa3091b13de 1169
lbzip2_2.2-1.dsc
4cb0e50f2fac1dfe12ab3bdc759abdc0227f9dacdd1bba507b37eecbe8666c8a 134040
lbzip2_2.2.orig.tar.gz
c4238ae4f9fd65dae16827b441fc2d7e8a251f3d1d39ba6360602a5a3dea070e 5352
lbzip2_2.2-1.debian.tar.bz2
7fcc9b05249b56b0a444e8c26812df86f7920c561a1ad065b223e432977acf7a 79960
lbzip2_2.2-1_amd64.deb
Files:
b324105c2f18959ff83454e3d258afe3 1169 utils extra lbzip2_2.2-1.dsc
8fe777a8405027ac33bf43095504d796 134040 utils extra lbzip2_2.2.orig.tar.gz
3e78576e933cf8cde911039ce7cef2a5 5352 utils extra lbzip2_2.2-1.debian.tar.bz2
46c7bd6d1850a82b1f6bf6323d56b7cd 79960 utils extra lbzip2_2.2-1_amd64.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iEYEARECAAYFAlAyDYcACgkQMDatjqUaT92kZACdFjTW6N5gLNjQQfxF63u/CgFh
AJsAn1mWBvzypoJUrJcLqj3UYVfe2UDx
=/VCd
-----END PGP SIGNATURE-----
--- End Message ---
