[RFC] zstd as a compression algorithm for LTO

2019-06-19 Thread Martin Liška
Hi. I've written a patch draft that replaces zlib with the zstd compression algorithm ([1]) in LTO. I'm also sending statistics that are collected for couple of quite big C++ source files. Observation I did: - LTO stream compression takes 3-4% of LGEN compile time - zstd in default compression

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-19 Thread Jeff Law
On 6/19/19 3:21 AM, Martin Liška wrote: > Hi. > > I've written a patch draft that replaces zlib with the zstd compression > algorithm ([1]) > in LTO. I'm also sending statistics that are collected for couple of quite > big C++ source > files. Observation I did: > > - LTO stream compression take

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-19 Thread Richard Biener
On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law wrote: >On 6/19/19 3:21 AM, Martin Liška wrote: >> Hi. >> >> I've written a patch draft that replaces zlib with the zstd >compression algorithm ([1]) >> in LTO. I'm also sending statistics that are collected for couple of >quite big C++ source >> fi

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-19 Thread Andrew Pinski
On Wed, Jun 19, 2019 at 11:55 AM Richard Biener wrote: > > On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law wrote: > >On 6/19/19 3:21 AM, Martin Liška wrote: > >> Hi. > >> > >> I've written a patch draft that replaces zlib with the zstd > >compression algorithm ([1]) > >> in LTO. I'm also sending

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-19 Thread Jan Hubicka
> > At least allow it to be built as part of the normal build like GMP, > etc. are done. > And include it in downloading using contrib/download_prerequisites > like the libraries are done. Anoying detail is that zstd builds with cmake, not autotools Honza > > Thanks, > Andrew Pinski > > > > >

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-19 Thread Andrew Pinski
On Wed, Jun 19, 2019 at 12:29 PM Jan Hubicka wrote: > > > > > At least allow it to be built as part of the normal build like GMP, > > etc. are done. > > And include it in downloading using contrib/download_prerequisites > > like the libraries are done. > > Anoying detail is that zstd builds with c

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Martin Liška
Hello. As mentioned by Honza, it's using cmake and to be honest I prefer to use a shared library than a statically build library. Moreover, it's an optional requirement and so that we don't have to include that to contrib/download_prerequisites. I like the idea of marking of compression algorit

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Segher Boessenkool
On Wed, Jun 19, 2019 at 09:29:54PM +0200, Jan Hubicka wrote: > > > > At least allow it to be built as part of the normal build like GMP, > > etc. are done. > > And include it in downloading using contrib/download_prerequisites > > like the libraries are done. > > Anoying detail is that zstd build

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Thomas Koenig
Am 20.06.19 um 11:07 schrieb Martin Liška: On the contrary, decompression of zstd with zlib will end with: lto1: internal compiler error: compressed stream: data error Sogenerating object files on one system and trying to read them on another system which does not happen to have a particular li

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Martin Liška
On 6/20/19 12:58 PM, Thomas Koenig wrote: > Am 20.06.19 um 11:07 schrieb Martin Liška: >> On the contrary, decompression >> of zstd with zlib will end with: >> lto1: internal compiler error: compressed stream: data error > > Sogenerating object files on one system and trying to read them > on anot

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Martin Liška
On 6/20/19 12:46 PM, Segher Boessenkool wrote: > OTOH, how well has this been tested, then? On different platforms, etc.? Dunno. The algorithm is quite new (4 years), but as it's becoming more popular. So I would expect coverage would be quite good. Maybe Nathan can comment on this? Martin

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Jan Hubicka
> On 6/20/19 12:58 PM, Thomas Koenig wrote: > > Am 20.06.19 um 11:07 schrieb Martin Liška: > >> On the contrary, decompression > >> of zstd with zlib will end with: > >> lto1: internal compiler error: compressed stream: data error > > > > Sogenerating object files on one system and trying to read

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Thomas Koenig
Hi Martin, LTO bytecode is not supposed to be a distributable format. One of my dreams is to make libgfortran LTO-clean. There is a lot of performance to be gained both in I/O (where a huge number of special cases could be shortcut by LTO, because hardly any program uses them all) and in arra

Re: [RFC] zstd as a compression algorithm for LTO

2019-06-20 Thread Joseph Myers
Any use of a host library should come with associated configure options to specify header and library paths for that library (and documentation for those options). (See existing --with-gmp*, --with-isl* etc. options.) -- Joseph S. Myers jos...@codesourcery.com