[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #1 from Oleg Endo  ---
Hmm ... if the program is changed in some places, it can also be triggered on
GCC 6.4.  It seems it happens when adding/removing function calls inside of C++
lambda functions ... how to find the cause?

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

Richard Biener  changed:

   What|Removed |Added

 CC||rguenth at gcc dot gnu.org

--- Comment #2 from Richard Biener  ---
It's a message from zlib:

  status = inflate (&in_stream, Z_SYNC_FLUSH);
  if (status != Z_OK && status != Z_STREAM_END)
internal_error ("compressed stream: %s", zError (status));

not sure what 'data error' means but I think the stream is corrupt.  Can you
try using valgrind on a reproducer to see if we have a wild memory write/read
somewhere?  And double-check your memory/disk?

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #3 from Oleg Endo  ---
BTW, I'm using -flto-compression-level=0.  In the past (GCC 4.5 ...) I've used
it as a workaround for other "lto compression related" errors.

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #4 from rguenther at suse dot de  ---
On Mon, 15 Jan 2018, olegendo at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816
> 
> --- Comment #3 from Oleg Endo  ---
> BTW, I'm using -flto-compression-level=0.  In the past (GCC 4.5 ...) I've used
> it as a workaround for other "lto compression related" errors.

I think it should either work always or never ...

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #5 from Oleg Endo  ---
(In reply to rguent...@suse.de from comment #4)
> 
> I think it should either work always or never ...

Yep.  In the past the compression-level setting showed different
success/failure rates.  In this case here, it seems consistent in this regard.

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

Richard Biener  changed:

   What|Removed |Added

   Target Milestone|--- |7.3

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #6 from Oleg Endo  ---
I can reproduce it on a different machine.  One of the object files in the
whole app build seems to be written bad.

I could isolate the pre-processed .ii file.  Compiling that .ii file and trying
to link it reproduces the above error.

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #7 from rguenther at suse dot de  ---
On January 15, 2018 5:04:14 PM GMT+01:00, "olegendo at gcc dot gnu.org"
 wrote:
>https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816
>
>--- Comment #6 from Oleg Endo  ---
>I can reproduce it on a different machine.  One of the object files in
>the
>whole app build seems to be written bad.
>
>I could isolate the pre-processed .ii file.  Compiling that .ii file
>and trying
>to link it reproduces the above error.

If you can produce a testcase and attach that that would be nice.

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #8 from Oleg Endo  ---
(In reply to rguent...@suse.de from comment #7)
> 
> If you can produce a testcase and attach that that would be nice.

I'm trying.  But it's gonna take ages.  Have to hand-strip the .ii file line by
line ... or else the error disappears.

I'm using system's zlib (version 1.2.8, stock ubuntu 16.04).  Could that be a
problem?

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #9 from Oleg Endo  ---
This is weird.  If I remove empty lines, or rename the paths in the # line
markers in the .ii file, the error sometimes disappears...

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #10 from rguenther at suse dot de  ---
On Mon, 15 Jan 2018, olegendo at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816
> 
> --- Comment #8 from Oleg Endo  ---
> (In reply to rguent...@suse.de from comment #7)
> > 
> > If you can produce a testcase and attach that that would be nice.
> 
> I'm trying.  But it's gonna take ages.  Have to hand-strip the .ii file line 
> by
> line ... or else the error disappears.
> 
> I'm using system's zlib (version 1.2.8, stock ubuntu 16.04).  Could that be a
> problem?

I don't think so.

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #11 from rguenther at suse dot de  ---
On Mon, 15 Jan 2018, olegendo at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816
> 
> --- Comment #9 from Oleg Endo  ---
> This is weird.  If I remove empty lines, or rename the paths in the # line
> markers in the .ii file, the error sometimes disappears...

Smells odd indeed.  Can you try building with 
--enable-valgrind-annotations and run valgrind on the thing?  My theory
would still be a wild write somewhere...

But to be honest the compression code is a mess ...

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #12 from Oleg Endo  ---
Created attachment 43152
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=43152&action=edit
preprocessed c++ source

I was able to reduce it somewhat.  However, I'd be surprised if it does not
reproduce the error on some other system.  I get the error only with the RX
cross compiler

rx-elf-gcc -v
Using built-in specs.
COLLECT_GCC=rx-elf-gcc
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/rx-elf/6.4.1/lto-wrapper
Target: rx-elf
Configured with: ../gcc-6-branch-rx/configure --target=rx-elf
--prefix=/usr/local --enable-languages=c,c++ --disable-nls --disable-werror
--with-newlib --enable-lto --enable-multilib --with-system-zlib
--disable-libstdcxx-verbose --disable-symvers
Thread model: single
gcc version 6.4.1 20180114 (GCC)

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #13 from Oleg Endo  ---
(In reply to Oleg Endo from comment #12)
> I was able to reduce it somewhat.  However, I'd be surprised if it does not
> reproduce the error on some other system.

I meant: I'd be surprised if it does reproduce the error .. still, would be
interesting to see if somebody else can reproduce it.

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[olegendo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

--- Comment #14 from Oleg Endo  ---
(In reply to rguent...@suse.de from comment #11)
> 
> Smells odd indeed.  Can you try building with 
> --enable-valgrind-annotations and run valgrind on the thing?  My theory
> would still be a wild write somewhere...

I can give it a try in a couple of days

[Bug lto/83816] [7 Regression] lto1: internal compiler error: compressed stream: data error

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83816

Richard Biener  changed:

   What|Removed |Added

   Keywords||lto
 Status|UNCONFIRMED |NEW
   Last reconfirmed||2018-01-17
 Ever confirmed|0   |1

--- Comment #15 from Richard Biener  ---
(In reply to Oleg Endo from comment #12)
> Created attachment 43152 [details]
> preprocessed c++ source
> 
> I was able to reduce it somewhat.  However, I'd be surprised if it does not
> reproduce the error on some other system.  I get the error only with the RX
> cross compiler
> 
> rx-elf-gcc -v
> Using built-in specs.
> COLLECT_GCC=rx-elf-gcc
> COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/rx-elf/6.4.1/lto-wrapper
> Target: rx-elf
> Configured with: ../gcc-6-branch-rx/configure --target=rx-elf
> --prefix=/usr/local --enable-languages=c,c++ --disable-nls --disable-werror
> --with-newlib --enable-lto --enable-multilib --with-system-zlib
> --disable-libstdcxx-verbose --disable-symvers
> Thread model: single
> gcc version 6.4.1 20180114 (GCC)

I can reproduce it:

rguenther@murzim:/abuild/rguenther/obj-rx-elf/gcc> ./xg++ -B. t.ii -flto
-nostdlib
/usr/bin/rx-elf-ld: warning: -z relro ignored.
lto1: internal compiler error: compressed stream: data error
0xa5a0ee lto_end_uncompression(lto_compression_stream*)
/space/rguenther/src/svn/gcc-6-branch/gcc/lto-compress.c:293
0xa57af9 lto_get_section_data(lto_file_decl_data*, lto_section_type, char
const*, unsigned long*, bool)
/space/rguenther/src/svn/gcc-6-branch/gcc/lto-section-in.c:164
0x6df605 lto_file_finalize
/space/rguenther/src/svn/gcc-6-branch/gcc/lto/lto.c:2038
0x6df694 lto_create_files_from_ids
/space/rguenther/src/svn/gcc-6-branch/gcc/lto/lto.c:2055
0x6df7bb lto_file_read
/space/rguenther/src/svn/gcc-6-branch/gcc/lto/lto.c:2096
0x6e1490 read_cgraph_and_symbols
/space/rguenther/src/svn/gcc-6-branch/gcc/lto/lto.c:2806
0x6e252f lto_main()
/space/rguenther/src/svn/gcc-6-branch/gcc/lto/lto.c:3310
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See  for instructions.
lto-wrapper: fatal error: ./xg++ returned 1 exit status
compilation terminated.
/usr/bin/rx-elf-ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

"Z_DATA_ERROR if the input data was
  corrupted (input stream not conforming to the zlib format or incorrect check
  value)"

We're reading LTO_section_decls at WPA time.

zlib runs into

for (;;)
switch (state->mode) {
case HEAD:
...
case BAD:
ret = Z_DATA_ERROR;
goto inf_leave;

caused by

case CHECK:
if (state->wrap) {
NEEDBITS(32);
out -= left;
strm->total_out += out;
state->total += out;
if (out)
strm->adler = state->check =
UPDATE(state->check, put - out, out);
out = left;
if ((
#ifdef GUNZIP
 state->flags ? hold :
#endif
 ZSWAP32(hold)) != state->check) {
strm->msg = (char *)"incorrect data check";
state->mode = BAD;
break;

where the checksums don't match.

I honestly have no idea what goes wrong ... :/

The above is with in-tree zlib and a cross built with -O0 -g so debugging this
is possible.  Just deflate/inflate is spread across cc1plus / lto1 invocations.

The compression/data code wasn't changed between GCC 6 and trunk so the issue
is probably present/latent in GCC 7 and GCC 8 as well.