Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic]

This still doesn't work.
All current versions, including cygwin 2.8.0-1.

Result same as in above testcase.
64 bit version works fine.

PS: Compressing the test.iso (or its already compressed version) in 32
bit environment with:
lrzip  -o doppel.lrz  test.iso(.lrz)
Gives:
Unable to malloc buffer of size 309956175 in flush_buffer
Fatal error - exiting

So in 32 bit, lrzip is seriously borken and should maybe be removed.


Regards,
David

On 5 February 2017 at 15:19, David Balažic  wrote:
> On 5 February 2017 at 07:37, Marco Atzeri  wrote:
>> can you check if latest cygwin test solves the issue ?
>
> I did, it doesn't, see my previous message in thread.
>
> David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic]

On 5 February 2017 at 07:37, Marco Atzeri  wrote:
> can you check if latest cygwin test solves the issue ?

I did, it doesn't, see my previous message in thread.

David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[Marco Atzeri]

On 25/01/2017 23:15, David Balažic wrote:

Hi!

The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the
decompressed dat in some circumstances.

I reproduced the problem on 2 PCs (the md5sum of the broken output was
the same on both systems).

I seems to happen when the (de)compressed file size is bigger than the
available RAM (note that the 32 bit version uses max 4GB in any case)
and lrzip resorts to using a temporary file.


[cut]



Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to
filesystem works fine, only when piped to stdout the problem happens.

The 64 bit version does not have this problem.






Regards,
David



can you check if latest cygwin test solves the issue ?
There was a change on pipe handling.

https://cygwin.com/ml/cygwin-announce/2017-02/msg7.html

- Always try to write all incoming bytes to blocking pipes, as required
  by POSIX.
  Addresses: https://cygwin.com/ml/cygwin/2017-01/msg00087.html

Regards
Marco


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic]

Here is a smaller reproducible test case (I did the preparation part
in 64 bit cygwin):

 - download http://releases.ubuntu.com/16.10/ubuntu-16.10-desktop-i386.iso
 - download http://releases.ubuntu.com/16.10/ubuntu-16.10-desktop-amd64.iso

$ md5sum ubuntu-16.10-desktop-i386.iso ubuntu-16.10-desktop-amd64.iso
e9e9a6c6b3c8c265788f4e726af25994 *ubuntu-16.10-desktop-i386.iso
3f50877c05121f7fd8544bef2d722824 *ubuntu-16.10-desktop-amd64.iso

$ cat ubuntu-16.10-desktop-i386.iso ubuntu-16.10-desktop-amd64.iso > test.iso

$ md5sum test.iso
d6e8cb13ce36e2c04961004782eec139 *test.iso

$ lrzip -v test.iso
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 4
Detected 17160601600 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Verbose
Temporary Directory set as: /tmp/
Compression mode is: LZMA. LZO Compressibility testing enabled
Heuristically Computed Compression Window: 109 = 10900MB
Output filename is: test.iso.lrz
File size: 3204448256
Will take 1 pass
Beginning rzip pre-processing phase
test.iso - Compression Ratio: 1.352. Average Compression Speed: 15.357MB/s.
Total time: 00:03:18.85

$ lrzip -i test.iso.lrz
test.iso.lrz:
lrzip version: 0.6 file
Compression: rzip + lzma
Decompressed file size: 3204448256
Compressed file size: 2370391407
Compression ratio: 1.352
MD5 used for integrity testing
MD5: d6e8cb13ce36e2c04961004782eec139

in 32 bit cygwin:

$ TMP=/cygdrive/d/tmp/  lrzip -v  -d  -o -
/cygdrive/c/cygwin64/home/stein/test.iso.lrz  | tee >(md5sum --tag)
>(sha1sum --tag) > /dev/null
The following options are in effect for this DECOMPRESSION.
Threading is ENABLED. Number of CPUs detected: 4
Detected 17160601600 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Verbose
Output Filename Specified: -
Temporary Directory set as: /cygdrive/d/tmp/
Outputting to stdout.
Detected lrzip version 0.6 file.
MD5 being used for integrity testing.
Decompressing...
Unable to decompress entirely in ram, will use physical files
Dumping temporary file to control->outFILE.

Average DeCompression Speed: 29.105MB/s
Dumping temporary file to control->outFILE.
[OK] - 3204448256 bytes
Total time: 00:01:45.07
SHA1 (-) = da39a3ee5e6b4b0d3255bfef95601890afd80709
MD5 (-) = d41d8cd98f00b204e9800998ecf8427e

The MD5 sum is wrong. (both md5 and sha1 values are the hashes of an
empty file, so the data is lost somewhere)

I attach the cygcheck -s -v -r output again (with cygwin 2.7.0-0.1 , I
later checked with 2.7.0-0.2 too, got the same results)

Regards,
David


On 1 February 2017 at 22:25, David Balažic  wrote:
> I tried the  2.7.0-0.1  test release and now the behavior changed.
>
> Before I got consistently a wrong MD5 sum of 8bd6ad48f2cea6a710af70b434d57673
> With this release I get c43a02c309fa5e0abe778201e9ceec46.
>
> So something changed.
>
> Either the problem is in cygwin or lrzip, I guess.
>
> Regards,
> David
>
>
> On 30 January 2017 at 16:23, David Balažic  wrote:
>> I tried in Ubuntu 32 bit (both the packaged lrzip  and a self compiled
>> one) and there the problem does not happen, so it looks like either:
>>  - bad lrzip in cygwin
>>  - cygwin pipe issues?
>>
>> Regards,
>> David
>>
>>
>> On 25 January 2017 at 23:15, David Balažic  wrote:
>>> Hi!
>>>
>>> The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the
>>> decompressed dat in some circumstances.
>>>
>>> I reproduced the problem on 2 PCs (the md5sum of the broken output was
>>> the same on both systems).
>>>
>>> I seems to happen when the (de)compressed file size is bigger than the
>>> available RAM (note that the 32 bit version uses max 4GB in any case)
>>> and lrzip resorts to using a temporary file.
>>>
>>> See below for reproducing:
>>>
>>> $ lrzip -i sda.img.lrz2
>>> sda.img.lrz2:
>>> lrzip version: 0.6 file
>>> Compression: rzip + lzma
>>> Decompressed file size: 64017212928
>>> Compressed file size: 7210541304
>>> Compression ratio: 8.878
>>> MD5 used for integrity testing
>>> MD5: 6594f5b0d22efd345003260054165842
>>>
>>> $ date; df -h ; TMP=/cygdrive/i/t/tmp/  lrzip -v  -d  -o -
>>> sda.img.lrz2  | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null   ;
>>> date
>>> Tue Jan 24 21:29:01 CET 2017
>>> Filesystem  Size  Used Avail Use% Mounted on
>>> C:/cygwin   114G   94G   21G  83% /
>>> D:  541G  534G  7.1G  99% /cygdrive/d
>>> I:  391G  279G  113G  72% /cygdrive/i
>>> Q:   60G   57G  2.8G  96% /cygdrive/q
>>> The following options are in effect for this DECOMPRESSION.
>>> Threading is ENABLED. Number of CPUs detected: 4
>>> Detected 17160601600 bytes ram
>>> Compression level 7
>>> Nice Value: 19
>>> Show Progress
>>> Verbose
>>> Output Filename Specified: -
>>> Temporary Directory set as: /cygdrive/i/t/tmp/
>>> Outputting to stdout.
>>> Detected lrzip version 0.6 file.
>>> MD5 being used for integrity testing.
>>> Decompressing...
>>> Unable to decompress entirely in ram, will use 

Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic]

I tried the  2.7.0-0.1  test release and now the behavior changed.

Before I got consistently a wrong MD5 sum of 8bd6ad48f2cea6a710af70b434d57673
With this release I get c43a02c309fa5e0abe778201e9ceec46.

So something changed.

Either the problem is in cygwin or lrzip, I guess.

Regards,
David


On 30 January 2017 at 16:23, David Balažic  wrote:
> I tried in Ubuntu 32 bit (both the packaged lrzip  and a self compiled
> one) and there the problem does not happen, so it looks like either:
>  - bad lrzip in cygwin
>  - cygwin pipe issues?
>
> Regards,
> David
>
>
> On 25 January 2017 at 23:15, David Balažic  wrote:
>> Hi!
>>
>> The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the
>> decompressed dat in some circumstances.
>>
>> I reproduced the problem on 2 PCs (the md5sum of the broken output was
>> the same on both systems).
>>
>> I seems to happen when the (de)compressed file size is bigger than the
>> available RAM (note that the 32 bit version uses max 4GB in any case)
>> and lrzip resorts to using a temporary file.
>>
>> See below for reproducing:
>>
>> $ lrzip -i sda.img.lrz2
>> sda.img.lrz2:
>> lrzip version: 0.6 file
>> Compression: rzip + lzma
>> Decompressed file size: 64017212928
>> Compressed file size: 7210541304
>> Compression ratio: 8.878
>> MD5 used for integrity testing
>> MD5: 6594f5b0d22efd345003260054165842
>>
>> $ date; df -h ; TMP=/cygdrive/i/t/tmp/  lrzip -v  -d  -o -
>> sda.img.lrz2  | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null   ;
>> date
>> Tue Jan 24 21:29:01 CET 2017
>> Filesystem  Size  Used Avail Use% Mounted on
>> C:/cygwin   114G   94G   21G  83% /
>> D:  541G  534G  7.1G  99% /cygdrive/d
>> I:  391G  279G  113G  72% /cygdrive/i
>> Q:   60G   57G  2.8G  96% /cygdrive/q
>> The following options are in effect for this DECOMPRESSION.
>> Threading is ENABLED. Number of CPUs detected: 4
>> Detected 17160601600 bytes ram
>> Compression level 7
>> Nice Value: 19
>> Show Progress
>> Verbose
>> Output Filename Specified: -
>> Temporary Directory set as: /cygdrive/i/t/tmp/
>> Outputting to stdout.
>> Detected lrzip version 0.6 file.
>> MD5 being used for integrity testing.
>> Decompressing...
>> Unable to decompress entirely in ram, will use physical files
>> Dumping temporary file to control->outFILE.
>>
>> [1]+  Stopped TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o -
>> sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null
>> Tue Jan 24 21:31:39 CET 2017
>>
>> stein@hofer8 /cygdrive/i/Zotac_bak
>> $ fg
>> TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum
>> --tag) >(sha1sum --tag) > /dev/null
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>> Dumping temporary file to control->outFILE.
>>
>> Average DeCompression Speed:  0.668MB/s
>> Dumping temporary file to control->outFILE.
>> [OK] - 64017212928 bytes
>> Total time: 25:22:26.25
>> SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72
>> MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673
>>
>>
>> The correct md5sum is 6594f5b0d22efd345003260054165842.
>>
>>
>> Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to
>> filesystem works fine, only when piped to stdout the problem happens.
>>
>> The 64 bit version does not have this problem.
>>
>>
>> I will check if the same problem happens with the native linux build
>> of lrzip (it takes a day...).
>>
>>
>> I tried to reproduce the problem with a smaller file, but there it did
>> not happen. Maybe my first test file has some corruption that causes
>> this (unlikely).
>>
>> Some version information (complete cygcheck -s -v -r output attached):
>>
>> base-cygwin   3.8-1
>> cygwin2.6.1-1
>> lrzip 0.631-1
>>
>> Regards,
>> David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic]

I tried in Ubuntu 32 bit (both the packaged lrzip  and a self compiled
one) and there the problem does not happen, so it looks like either:
 - bad lrzip in cygwin
 - cygwin pipe issues?

Regards,
David


On 25 January 2017 at 23:15, David Balažic  wrote:
> Hi!
>
> The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the
> decompressed dat in some circumstances.
>
> I reproduced the problem on 2 PCs (the md5sum of the broken output was
> the same on both systems).
>
> I seems to happen when the (de)compressed file size is bigger than the
> available RAM (note that the 32 bit version uses max 4GB in any case)
> and lrzip resorts to using a temporary file.
>
> See below for reproducing:
>
> $ lrzip -i sda.img.lrz2
> sda.img.lrz2:
> lrzip version: 0.6 file
> Compression: rzip + lzma
> Decompressed file size: 64017212928
> Compressed file size: 7210541304
> Compression ratio: 8.878
> MD5 used for integrity testing
> MD5: 6594f5b0d22efd345003260054165842
>
> $ date; df -h ; TMP=/cygdrive/i/t/tmp/  lrzip -v  -d  -o -
> sda.img.lrz2  | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null   ;
> date
> Tue Jan 24 21:29:01 CET 2017
> Filesystem  Size  Used Avail Use% Mounted on
> C:/cygwin   114G   94G   21G  83% /
> D:  541G  534G  7.1G  99% /cygdrive/d
> I:  391G  279G  113G  72% /cygdrive/i
> Q:   60G   57G  2.8G  96% /cygdrive/q
> The following options are in effect for this DECOMPRESSION.
> Threading is ENABLED. Number of CPUs detected: 4
> Detected 17160601600 bytes ram
> Compression level 7
> Nice Value: 19
> Show Progress
> Verbose
> Output Filename Specified: -
> Temporary Directory set as: /cygdrive/i/t/tmp/
> Outputting to stdout.
> Detected lrzip version 0.6 file.
> MD5 being used for integrity testing.
> Decompressing...
> Unable to decompress entirely in ram, will use physical files
> Dumping temporary file to control->outFILE.
>
> [1]+  Stopped TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o -
> sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null
> Tue Jan 24 21:31:39 CET 2017
>
> stein@hofer8 /cygdrive/i/Zotac_bak
> $ fg
> TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum
> --tag) >(sha1sum --tag) > /dev/null
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
> Dumping temporary file to control->outFILE.
>
> Average DeCompression Speed:  0.668MB/s
> Dumping temporary file to control->outFILE.
> [OK] - 64017212928 bytes
> Total time: 25:22:26.25
> SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72
> MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673
>
>
> The correct md5sum is 6594f5b0d22efd345003260054165842.
>
>
> Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to
> filesystem works fine, only when piped to stdout the problem happens.
>
> The 64 bit version does not have this problem.
>
>
> I will check if the same problem happens with the native linux build
> of lrzip (it takes a day...).
>
>
> I tried to reproduce the problem with a smaller file, but there it did
> not happen. Maybe my first test file has some corruption that causes
> this (unlikely).
>
> Some version information (complete cygcheck -s -v -r output attached):
>
> base-cygwin   3.8-1
> cygwin2.6.1-1
> lrzip 0.631-1
>
> Regards,
> David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic]

Hi!

The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the
decompressed dat in some circumstances.

I reproduced the problem on 2 PCs (the md5sum of the broken output was
the same on both systems).

I seems to happen when the (de)compressed file size is bigger than the
available RAM (note that the 32 bit version uses max 4GB in any case)
and lrzip resorts to using a temporary file.

See below for reproducing:

$ lrzip -i sda.img.lrz2
sda.img.lrz2:
lrzip version: 0.6 file
Compression: rzip + lzma
Decompressed file size: 64017212928
Compressed file size: 7210541304
Compression ratio: 8.878
MD5 used for integrity testing
MD5: 6594f5b0d22efd345003260054165842

$ date; df -h ; TMP=/cygdrive/i/t/tmp/  lrzip -v  -d  -o -
sda.img.lrz2  | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null   ;
date
Tue Jan 24 21:29:01 CET 2017
Filesystem  Size  Used Avail Use% Mounted on
C:/cygwin   114G   94G   21G  83% /
D:  541G  534G  7.1G  99% /cygdrive/d
I:  391G  279G  113G  72% /cygdrive/i
Q:   60G   57G  2.8G  96% /cygdrive/q
The following options are in effect for this DECOMPRESSION.
Threading is ENABLED. Number of CPUs detected: 4
Detected 17160601600 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Verbose
Output Filename Specified: -
Temporary Directory set as: /cygdrive/i/t/tmp/
Outputting to stdout.
Detected lrzip version 0.6 file.
MD5 being used for integrity testing.
Decompressing...
Unable to decompress entirely in ram, will use physical files
Dumping temporary file to control->outFILE.

[1]+  Stopped TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o -
sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null
Tue Jan 24 21:31:39 CET 2017

stein@hofer8 /cygdrive/i/Zotac_bak
$ fg
TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum
--tag) >(sha1sum --tag) > /dev/null
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.
Dumping temporary file to control->outFILE.

Average DeCompression Speed:  0.668MB/s
Dumping temporary file to control->outFILE.
[OK] - 64017212928 bytes
Total time: 25:22:26.25
SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72
MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673


The correct md5sum is 6594f5b0d22efd345003260054165842.


Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to
filesystem works fine, only when piped to stdout the problem happens.

The 64 bit version does not have this problem.


I will check if the same problem happens with the native linux build
of lrzip (it takes a day...).


I tried to reproduce the problem with a smaller file, but there it did
not happen. Maybe my first test file has some corruption that causes
this (unlikely).

Some version information (complete cygcheck -s -v -r output attached):

base-cygwin   3.8-1
cygwin2.6.1-1
lrzip 0.631-1

Regards,
David


cygcheck.out
Description: Binary data

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple