Your message dated Fri, 29 Apr 2011 03:33:05 +0000
with message-id <[email protected]>
and subject line Bug#623745: fixed in lrzip 0.603+2011.0423+git7ed977b-1
has caused the Debian Bug report #623745,
regarding lrzip: Fails to compress big enough (tens or hundreds of GB) because
of a thread leak
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
623745: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623745
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: lrzip
Version: 0.552+20110217+gitcd8b086-1
Severity: normal
[NOTE: This still seems to be a problem with the most recent upstream
version of 0.603 - try with "lrzip -lvv" - hence Cc:ing to upstream
Con Kolivas.]
Hi,
While trying to compress a big (172 GiB) file (with lrzip -nvv), I ran
into this error:
------------------------------------------------------------
Starting thread 7 to compress 19381057 bytes from stream 1
Starting thread 0 to compress 19381057 bytes from stream 1
pthread_createResource temporarily unavailable
Fatal error - exiting
------------------------------------------------------------
It turns out lrzip does not pthread_join() the threads it creates,
which in turn eventually exhausts the threads. You can verify this by
modifying it to print some debug information:
------------------------------------------------------------
--- lrzip-0.552+20110217+gitcd8b086/stream.c 2011-02-17 00:32:01.000000000
+0200
+++ lrzip.mod/stream.c 2011-04-22 19:16:04.000000000 +0300
@@ -71,15 +71,21 @@
fatal("pthread_cond_broadcast failed");
}
+static int threads_created=0, threads_joined=0;
+
void create_pthread(pthread_t * thread, pthread_attr_t * attr,
void * (*start_routine)(void *), void *arg)
{
+ print_maxverbose("create_pthread: created %d, joined %d\n",
+ ++threads_created, threads_joined);
if (pthread_create(thread, attr, start_routine, arg))
fatal("pthread_create");
}
void join_pthread(pthread_t th, void **thread_return)
{
+ print_maxverbose("join_pthread: created %d, joined %d\n",
+ threads_created, ++threads_joined);
if (pthread_join(th, thread_return))
fatal("pthread_join");
}
------------------------------------------------------------
Now it is evident that join_pthread() is not called (and there is no
other place in the source that would call pthread_join()):
------------------------------------------------------------
$ lrzip -nvv root.dump
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 8
Detected 8362065920 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Max Verbose
Compression mode is: RZIP pre-processing only
Output filename is: root.dump.lrz
File size: 184045373440
Enabling sliding mmap mode and using mmap of 2787352576 bytes with window of
5574709248 bytes
Succeeded in testing 2787352576 sized mmap for rzip pre-processing
Will take 34 passes
Chunk size: 5574709248
Byte width: 5
Succeeded in testing 3498805584 sized malloc for back end compression
Using 8 threads to compress up to 218675349 bytes each.
Beginning rzip pre-processing phase
hashsize = 4194304. bits = 22. 64MB
Starting sweep for mask 1
Starting sweep for mask 3
Starting sweep for mask 7
Starting sweep for mask 15
Starting sweep for mask 31
Starting sweep for mask 63
Starting thread 0 to compress 218675349 bytes from stream 1
create_pthread: created 1, joined 0
Thread 0 writing 218675349 compressed bytes from stream 1
Starting thread 1 to compress 218675349 bytes from stream 1
create_pthread: created 2, joined 0
Thread 1 writing 218675349 compressed bytes from stream 1
Starting sweep for mask 127
Starting thread 2 to compress 218675349 bytes from stream 1
create_pthread: created 3, joined 0
Thread 2 writing 218675349 compressed bytes from stream 1
Starting thread 3 to compress 218675349 bytes from stream 1
create_pthread: created 4, joined 0
Thread 3 writing 218675349 compressed bytes from stream 1
Starting thread 4 to compress 218675349 bytes from stream 1
create_pthread: created 5, joined 0
Thread 4 writing 218675349 compressed bytes from stream 1
Starting sweep for mask 255
Starting thread 5 to compress 218675349 bytes from stream 1
create_pthread: created 6, joined 0
Thread 5 writing 218675349 compressed bytes from stream 1
Starting thread 6 to compress 218675349 bytes from stream 1
create_pthread: created 7, joined 0
Thread 6 writing 218675349 compressed bytes from stream 1
Starting thread 7 to compress 218675349 bytes from stream 1
create_pthread: created 8, joined 0
Thread 7 writing 218675349 compressed bytes from stream 1
Sliding main buffer to offset 2787352576
Starting thread 0 to compress 218675349 bytes from stream 1
create_pthread: created 9, joined 0
Thread 0 writing 218675349 compressed bytes from stream 1
Starting thread 1 to compress 218675349 bytes from stream 1
create_pthread: created 10, joined 0
Thread 1 writing 218675349 compressed bytes from stream 1
Starting thread 2 to compress 218675349 bytes from stream 1
create_pthread: created 11, joined 0
[...]
Pass 5 / 34 -- Elapsed Time: 00:17:24. ETA: 02:06:7092. Compress Speed:
20.766MB/s.
Succeeded in testing 2834032518 sized malloc for back end compression
Using 8 threads to compress up to 177127033 bytes each.
Beginning rzip pre-processing phase
Thread 4 writing 10575901 compressed bytes from stream 0
Thread 5 writing 176603041 compressed bytes from stream 1
Starting sweep for mask 1
Starting sweep for mask 3
Starting sweep for mask 7
Starting sweep for mask 15
Starting sweep for mask 31
Starting thread 6 to compress 177127033 bytes from stream 1
create_pthread: created 55, joined 0
Thread 6 writing 177127033 compressed bytes from stream 1
Starting sweep for mask 63
Starting thread 7 to compress 177127033 bytes from stream 1
create_pthread: created 56, joined 0
Thread 7 writing 177127033 compressed bytes from stream 1
[...]
------------------------------------------------------------
Sami
-- System Information:
Debian Release: wheezy/sid
APT prefers oldstable
APT policy: (500, 'oldstable'), (500, 'unstable'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.38.4 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages lrzip depends on:
ii bash 4.1-3 The GNU Bourne Again SHell
ii libbz2-1.0 1.0.5-6 high-quality block-sorting file co
ii libc6 2.11.2-13 Embedded GNU C Library: Shared lib
ii libgcc1 1:4.6.0-4 GCC support library
ii liblzo2-2 2.04-1 data compression library
ii libstdc++6 4.6.0-4 The GNU Standard C++ Library v3
ii zlib1g 1:1.2.3.4.dfsg-3 compression library - runtime
lrzip recommends no packages.
lrzip suggests no packages.
-- no debconf information
signature.asc
Description: Digital signature
--- End Message ---
--- Begin Message ---
Source: lrzip
Source-Version: 0.603+2011.0423+git7ed977b-1
We believe that the bug you reported is fixed in the latest version of
lrzip, which is due to be installed in the Debian FTP archive:
lrzip_0.603+2011.0423+git7ed977b-1.debian.tar.gz
to main/l/lrzip/lrzip_0.603+2011.0423+git7ed977b-1.debian.tar.gz
lrzip_0.603+2011.0423+git7ed977b-1.dsc
to main/l/lrzip/lrzip_0.603+2011.0423+git7ed977b-1.dsc
lrzip_0.603+2011.0423+git7ed977b-1_i386.deb
to main/l/lrzip/lrzip_0.603+2011.0423+git7ed977b-1_i386.deb
lrzip_0.603+2011.0423+git7ed977b.orig.tar.gz
to main/l/lrzip/lrzip_0.603+2011.0423+git7ed977b.orig.tar.gz
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Jari Aalto <[email protected]> (supplier of updated lrzip package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Tue, 26 Apr 2011 10:30:59 +0300
Source: lrzip
Binary: lrzip
Architecture: source i386
Version: 0.603+2011.0423+git7ed977b-1
Distribution: unstable
Urgency: low
Maintainer: Jari Aalto <[email protected]>
Changed-By: Jari Aalto <[email protected]>
Description:
lrzip - compression program with a very high compression ratio
Closes: 623745
Changes:
lrzip (0.603+2011.0423+git7ed977b-1) unstable; urgency=low
.
* New upstream snapshot.
- Fix failure to compress big files (Closes: #623745).
* debian/control
- (Build-Depends): Add automake, autoconf, libtool.
* debian/rules
- (override_dh_auto_configure): New. For snapshot packaging.
Checksums-Sha1:
55ec49b6c128d10233c828467a048db091b2ad6d 1984
lrzip_0.603+2011.0423+git7ed977b-1.dsc
e58aab1e17dad99408dc992e4999e38cb2e597d1 188338
lrzip_0.603+2011.0423+git7ed977b.orig.tar.gz
e1e3032c77c481209356b9a00487ac870c22b931 7046
lrzip_0.603+2011.0423+git7ed977b-1.debian.tar.gz
8f12a580a4ef325ddc0eb30f7ca2c10a6a1b403f 167580
lrzip_0.603+2011.0423+git7ed977b-1_i386.deb
Checksums-Sha256:
cebd5d1c54f5ee8a05c2ca97de3be61da62ceb1fe18aec182b80321fe5e55341 1984
lrzip_0.603+2011.0423+git7ed977b-1.dsc
c934a406176a633e7968a04edc0bd8f1243ce5b81d3f1314759a10f42b579d11 188338
lrzip_0.603+2011.0423+git7ed977b.orig.tar.gz
532bdfa471a11ec4b289d8b52c54259ff6e25b483cc1d8086bcf7f388640410e 7046
lrzip_0.603+2011.0423+git7ed977b-1.debian.tar.gz
471380c4abac1857ce994fcbba09593bef4cb1887aafc469e3e3373231eb5117 167580
lrzip_0.603+2011.0423+git7ed977b-1_i386.deb
Files:
f8ee0f7dffce05051eec459780d114f6 1984 utils optional
lrzip_0.603+2011.0423+git7ed977b-1.dsc
3952e50fa1c08761d03af5a0118eca47 188338 utils optional
lrzip_0.603+2011.0423+git7ed977b.orig.tar.gz
2ce4941fb17392a86de55a7a580fcc0f 7046 utils optional
lrzip_0.603+2011.0423+git7ed977b-1.debian.tar.gz
effa2648532bebde9af60f657954ef03 167580 utils optional
lrzip_0.603+2011.0423+git7ed977b-1_i386.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iQIcBAEBCAAGBQJNuipuAAoJECHSBYmXSz6Wn4cP/37CNI0OBtt3dhWg5RNqaBbu
c3SgUiJcFSgAZNibSvn69n2WH9hM9REjbLHeUrLHCcRPkpozECkC21CJqejy2kTG
sQSsvW+ieB1CwXliPEjSyTnXDXCtPNJOB4Otaudd+LzHPwiRKMH9GbFrGe3ZUOpU
1Y+qZZwJQFzDEwbkzXMH7inB0Ho5H3c/OsKhJljotzF2zitLFpXNqj30WXLArpMA
rpK+EVvq84D7vDwtwvV1i6o/5Qo4BaycYZn8t1/NIWi26k4B+KAottihPUzAqKd+
0qmpbw+XpviBLJaVMZ6BxO9OCHvAJC6049zr5bY2+iTPqtNaTQhkWIk5E1unfTZO
Terq44i4LvhEmDW15S0Ca+n22EwNkYiIQizaU/PLsSI2Vzf5rSTAPg23WnkIhaOV
VEX1n8wr96a4+VRcoomIpuOx9e641qlZuuLR5np2D1mgkJSxU8M4VBZ10AalMSky
XMampItISSq7Lsl89lCpLGIfiPISD45gC+FcirEEApAZnfA+l0saRl/DIsQ9LMHc
7zY/tlDRTyitJj9fDbC/7m13DS06koExFCiBTxc73hwPkxXkqsS0ASAoJK/djxfu
Lh+Cx9KJIiGzcer3kt9Q3MhwniX+moUTKWOStYKo6+bzG6k+WMcmuGAbqCy/S9lH
3WiSUySWBwJK7N8bvXiN
=Hssu
-----END PGP SIGNATURE-----
--- End Message ---
