Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
This still doesn't work. All current versions, including cygwin 2.8.0-1. Result same as in above testcase. 64 bit version works fine. PS: Compressing the test.iso (or its already compressed version) in 32 bit environment with: lrzip -o doppel.lrz test.iso(.lrz) Gives: Unable to malloc buffer of size 309956175 in flush_buffer Fatal error - exiting So in 32 bit, lrzip is seriously borken and should maybe be removed. Regards, David On 5 February 2017 at 15:19, David Balažic wrote: > On 5 February 2017 at 07:37, Marco Atzeri wrote: >> can you check if latest cygwin test solves the issue ? > > I did, it doesn't, see my previous message in thread. > > David -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
On 5 February 2017 at 07:37, Marco Atzeri wrote: > can you check if latest cygwin test solves the issue ? I did, it doesn't, see my previous message in thread. David -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
On 25/01/2017 23:15, David Balažic wrote: Hi! The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the decompressed dat in some circumstances. I reproduced the problem on 2 PCs (the md5sum of the broken output was the same on both systems). I seems to happen when the (de)compressed file size is bigger than the available RAM (note that the 32 bit version uses max 4GB in any case) and lrzip resorts to using a temporary file. [cut] Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to filesystem works fine, only when piped to stdout the problem happens. The 64 bit version does not have this problem. Regards, David can you check if latest cygwin test solves the issue ? There was a change on pipe handling. https://cygwin.com/ml/cygwin-announce/2017-02/msg7.html - Always try to write all incoming bytes to blocking pipes, as required by POSIX. Addresses: https://cygwin.com/ml/cygwin/2017-01/msg00087.html Regards Marco -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
Here is a smaller reproducible test case (I did the preparation part in 64 bit cygwin): - download http://releases.ubuntu.com/16.10/ubuntu-16.10-desktop-i386.iso - download http://releases.ubuntu.com/16.10/ubuntu-16.10-desktop-amd64.iso $ md5sum ubuntu-16.10-desktop-i386.iso ubuntu-16.10-desktop-amd64.iso e9e9a6c6b3c8c265788f4e726af25994 *ubuntu-16.10-desktop-i386.iso 3f50877c05121f7fd8544bef2d722824 *ubuntu-16.10-desktop-amd64.iso $ cat ubuntu-16.10-desktop-i386.iso ubuntu-16.10-desktop-amd64.iso > test.iso $ md5sum test.iso d6e8cb13ce36e2c04961004782eec139 *test.iso $ lrzip -v test.iso The following options are in effect for this COMPRESSION. Threading is ENABLED. Number of CPUs detected: 4 Detected 17160601600 bytes ram Compression level 7 Nice Value: 19 Show Progress Verbose Temporary Directory set as: /tmp/ Compression mode is: LZMA. LZO Compressibility testing enabled Heuristically Computed Compression Window: 109 = 10900MB Output filename is: test.iso.lrz File size: 3204448256 Will take 1 pass Beginning rzip pre-processing phase test.iso - Compression Ratio: 1.352. Average Compression Speed: 15.357MB/s. Total time: 00:03:18.85 $ lrzip -i test.iso.lrz test.iso.lrz: lrzip version: 0.6 file Compression: rzip + lzma Decompressed file size: 3204448256 Compressed file size: 2370391407 Compression ratio: 1.352 MD5 used for integrity testing MD5: d6e8cb13ce36e2c04961004782eec139 in 32 bit cygwin: $ TMP=/cygdrive/d/tmp/ lrzip -v -d -o - /cygdrive/c/cygwin64/home/stein/test.iso.lrz | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null The following options are in effect for this DECOMPRESSION. Threading is ENABLED. Number of CPUs detected: 4 Detected 17160601600 bytes ram Compression level 7 Nice Value: 19 Show Progress Verbose Output Filename Specified: - Temporary Directory set as: /cygdrive/d/tmp/ Outputting to stdout. Detected lrzip version 0.6 file. MD5 being used for integrity testing. Decompressing... Unable to decompress entirely in ram, will use physical files Dumping temporary file to control->outFILE. Average DeCompression Speed: 29.105MB/s Dumping temporary file to control->outFILE. [OK] - 3204448256 bytes Total time: 00:01:45.07 SHA1 (-) = da39a3ee5e6b4b0d3255bfef95601890afd80709 MD5 (-) = d41d8cd98f00b204e9800998ecf8427e The MD5 sum is wrong. (both md5 and sha1 values are the hashes of an empty file, so the data is lost somewhere) I attach the cygcheck -s -v -r output again (with cygwin 2.7.0-0.1 , I later checked with 2.7.0-0.2 too, got the same results) Regards, David On 1 February 2017 at 22:25, David Balažic wrote: > I tried the 2.7.0-0.1 test release and now the behavior changed. > > Before I got consistently a wrong MD5 sum of 8bd6ad48f2cea6a710af70b434d57673 > With this release I get c43a02c309fa5e0abe778201e9ceec46. > > So something changed. > > Either the problem is in cygwin or lrzip, I guess. > > Regards, > David > > > On 30 January 2017 at 16:23, David Balažic wrote: >> I tried in Ubuntu 32 bit (both the packaged lrzip and a self compiled >> one) and there the problem does not happen, so it looks like either: >> - bad lrzip in cygwin >> - cygwin pipe issues? >> >> Regards, >> David >> >> >> On 25 January 2017 at 23:15, David Balažic wrote: >>> Hi! >>> >>> The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the >>> decompressed dat in some circumstances. >>> >>> I reproduced the problem on 2 PCs (the md5sum of the broken output was >>> the same on both systems). >>> >>> I seems to happen when the (de)compressed file size is bigger than the >>> available RAM (note that the 32 bit version uses max 4GB in any case) >>> and lrzip resorts to using a temporary file. >>> >>> See below for reproducing: >>> >>> $ lrzip -i sda.img.lrz2 >>> sda.img.lrz2: >>> lrzip version: 0.6 file >>> Compression: rzip + lzma >>> Decompressed file size: 64017212928 >>> Compressed file size: 7210541304 >>> Compression ratio: 8.878 >>> MD5 used for integrity testing >>> MD5: 6594f5b0d22efd345003260054165842 >>> >>> $ date; df -h ; TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - >>> sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null ; >>> date >>> Tue Jan 24 21:29:01 CET 2017 >>> Filesystem Size Used Avail Use% Mounted on >>> C:/cygwin 114G 94G 21G 83% / >>> D: 541G 534G 7.1G 99% /cygdrive/d >>> I: 391G 279G 113G 72% /cygdrive/i >>> Q: 60G 57G 2.8G 96% /cygdrive/q >>> The following options are in effect for this DECOMPRESSION. >>> Threading is ENABLED. Number of CPUs detected: 4 >>> Detected 17160601600 bytes ram >>> Compression level 7 >>> Nice Value: 19 >>> Show Progress >>> Verbose >>> Output Filename Specified: - >>> Temporary Directory set as: /cygdrive/i/t/tmp/ >>> Outputting to stdout. >>> Detected lrzip version 0.6 file. >>> MD5 being used for integrity testing. >>> Decompressing... >>> Unable to decompress entirely in ram, will use physical files >>> Dumping temporary file to control->outFIL
Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
I tried the 2.7.0-0.1 test release and now the behavior changed. Before I got consistently a wrong MD5 sum of 8bd6ad48f2cea6a710af70b434d57673 With this release I get c43a02c309fa5e0abe778201e9ceec46. So something changed. Either the problem is in cygwin or lrzip, I guess. Regards, David On 30 January 2017 at 16:23, David Balažic wrote: > I tried in Ubuntu 32 bit (both the packaged lrzip and a self compiled > one) and there the problem does not happen, so it looks like either: > - bad lrzip in cygwin > - cygwin pipe issues? > > Regards, > David > > > On 25 January 2017 at 23:15, David Balažic wrote: >> Hi! >> >> The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the >> decompressed dat in some circumstances. >> >> I reproduced the problem on 2 PCs (the md5sum of the broken output was >> the same on both systems). >> >> I seems to happen when the (de)compressed file size is bigger than the >> available RAM (note that the 32 bit version uses max 4GB in any case) >> and lrzip resorts to using a temporary file. >> >> See below for reproducing: >> >> $ lrzip -i sda.img.lrz2 >> sda.img.lrz2: >> lrzip version: 0.6 file >> Compression: rzip + lzma >> Decompressed file size: 64017212928 >> Compressed file size: 7210541304 >> Compression ratio: 8.878 >> MD5 used for integrity testing >> MD5: 6594f5b0d22efd345003260054165842 >> >> $ date; df -h ; TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - >> sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null ; >> date >> Tue Jan 24 21:29:01 CET 2017 >> Filesystem Size Used Avail Use% Mounted on >> C:/cygwin 114G 94G 21G 83% / >> D: 541G 534G 7.1G 99% /cygdrive/d >> I: 391G 279G 113G 72% /cygdrive/i >> Q: 60G 57G 2.8G 96% /cygdrive/q >> The following options are in effect for this DECOMPRESSION. >> Threading is ENABLED. Number of CPUs detected: 4 >> Detected 17160601600 bytes ram >> Compression level 7 >> Nice Value: 19 >> Show Progress >> Verbose >> Output Filename Specified: - >> Temporary Directory set as: /cygdrive/i/t/tmp/ >> Outputting to stdout. >> Detected lrzip version 0.6 file. >> MD5 being used for integrity testing. >> Decompressing... >> Unable to decompress entirely in ram, will use physical files >> Dumping temporary file to control->outFILE. >> >> [1]+ Stopped TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - >> sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null >> Tue Jan 24 21:31:39 CET 2017 >> >> stein@hofer8 /cygdrive/i/Zotac_bak >> $ fg >> TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum >> --tag) >(sha1sum --tag) > /dev/null >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> Dumping temporary file to control->outFILE. >> >> Average DeCompression Speed: 0.668MB/s >> Dumping temporary file to control->outFILE. >> [OK] - 64017212928 bytes >> Total time: 25:22:26.25 >> SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72 >> MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673 >> >> >> The correct md5sum is 6594f5b0d22efd345003260054165842. >> >> >> Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to >> filesystem works fine, only when piped to stdout the problem happens. >> >> The 64 bit version does not have this problem. >> >> >> I will check if the same problem happens with the native linux build >> of lrzip (it takes a day...). >> >> >> I tried to reproduce the problem with a smaller file, but there it did >> not happen. Maybe my first test file has some corruption that causes >> this (unlikely). >> >> Some version information (complete cygcheck -s -v -r output attached): >> >> base-cygwin 3.8-1 >> cygwin2.6.1-1 >> lrzip 0.631-1 >> >> Regards, >> David -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
I tried in Ubuntu 32 bit (both the packaged lrzip and a self compiled one) and there the problem does not happen, so it looks like either: - bad lrzip in cygwin - cygwin pipe issues? Regards, David On 25 January 2017 at 23:15, David Balažic wrote: > Hi! > > The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the > decompressed dat in some circumstances. > > I reproduced the problem on 2 PCs (the md5sum of the broken output was > the same on both systems). > > I seems to happen when the (de)compressed file size is bigger than the > available RAM (note that the 32 bit version uses max 4GB in any case) > and lrzip resorts to using a temporary file. > > See below for reproducing: > > $ lrzip -i sda.img.lrz2 > sda.img.lrz2: > lrzip version: 0.6 file > Compression: rzip + lzma > Decompressed file size: 64017212928 > Compressed file size: 7210541304 > Compression ratio: 8.878 > MD5 used for integrity testing > MD5: 6594f5b0d22efd345003260054165842 > > $ date; df -h ; TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - > sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null ; > date > Tue Jan 24 21:29:01 CET 2017 > Filesystem Size Used Avail Use% Mounted on > C:/cygwin 114G 94G 21G 83% / > D: 541G 534G 7.1G 99% /cygdrive/d > I: 391G 279G 113G 72% /cygdrive/i > Q: 60G 57G 2.8G 96% /cygdrive/q > The following options are in effect for this DECOMPRESSION. > Threading is ENABLED. Number of CPUs detected: 4 > Detected 17160601600 bytes ram > Compression level 7 > Nice Value: 19 > Show Progress > Verbose > Output Filename Specified: - > Temporary Directory set as: /cygdrive/i/t/tmp/ > Outputting to stdout. > Detected lrzip version 0.6 file. > MD5 being used for integrity testing. > Decompressing... > Unable to decompress entirely in ram, will use physical files > Dumping temporary file to control->outFILE. > > [1]+ Stopped TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - > sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null > Tue Jan 24 21:31:39 CET 2017 > > stein@hofer8 /cygdrive/i/Zotac_bak > $ fg > TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum > --tag) >(sha1sum --tag) > /dev/null > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > Dumping temporary file to control->outFILE. > > Average DeCompression Speed: 0.668MB/s > Dumping temporary file to control->outFILE. > [OK] - 64017212928 bytes > Total time: 25:22:26.25 > SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72 > MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673 > > > The correct md5sum is 6594f5b0d22efd345003260054165842. > > > Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to > filesystem works fine, only when piped to stdout the problem happens. > > The 64 bit version does not have this problem. > > > I will check if the same problem happens with the native linux build > of lrzip (it takes a day...). > > > I tried to reproduce the problem with a smaller file, but there it did > not happen. Maybe my first test file has some corruption that causes > this (unlikely). > > Some version information (complete cygcheck -s -v -r output attached): > > base-cygwin 3.8-1 > cygwin2.6.1-1 > lrzip 0.631-1 > > Regards, > David -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug in lrzip 0.631-1 (32 bit version) with -d -o - options
Hi! The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the decompressed dat in some circumstances. I reproduced the problem on 2 PCs (the md5sum of the broken output was the same on both systems). I seems to happen when the (de)compressed file size is bigger than the available RAM (note that the 32 bit version uses max 4GB in any case) and lrzip resorts to using a temporary file. See below for reproducing: $ lrzip -i sda.img.lrz2 sda.img.lrz2: lrzip version: 0.6 file Compression: rzip + lzma Decompressed file size: 64017212928 Compressed file size: 7210541304 Compression ratio: 8.878 MD5 used for integrity testing MD5: 6594f5b0d22efd345003260054165842 $ date; df -h ; TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null ; date Tue Jan 24 21:29:01 CET 2017 Filesystem Size Used Avail Use% Mounted on C:/cygwin 114G 94G 21G 83% / D: 541G 534G 7.1G 99% /cygdrive/d I: 391G 279G 113G 72% /cygdrive/i Q: 60G 57G 2.8G 96% /cygdrive/q The following options are in effect for this DECOMPRESSION. Threading is ENABLED. Number of CPUs detected: 4 Detected 17160601600 bytes ram Compression level 7 Nice Value: 19 Show Progress Verbose Output Filename Specified: - Temporary Directory set as: /cygdrive/i/t/tmp/ Outputting to stdout. Detected lrzip version 0.6 file. MD5 being used for integrity testing. Decompressing... Unable to decompress entirely in ram, will use physical files Dumping temporary file to control->outFILE. [1]+ Stopped TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null Tue Jan 24 21:31:39 CET 2017 stein@hofer8 /cygdrive/i/Zotac_bak $ fg TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Dumping temporary file to control->outFILE. Average DeCompression Speed: 0.668MB/s Dumping temporary file to control->outFILE. [OK] - 64017212928 bytes Total time: 25:22:26.25 SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72 MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673 The correct md5sum is 6594f5b0d22efd345003260054165842. Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to filesystem works fine, only when piped to stdout the problem happens. The 64 bit version does not have this problem. I will check if the same problem happens with the native linux build of lrzip (it takes a day...). I tried to reproduce the problem with a smaller file, but there it did not happen. Maybe my first test file has some corruption that causes this (unlikely). Some version information (complete cygcheck -s -v -r output attached): base-cygwin 3.8-1 cygwin2.6.1-1 lrzip 0.631-1 Regards, David cygcheck.out Description: Binary data -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple