Re: [ccache] Compression

[Joel Rosdahl via ccache]

Hi Francios,

On Tue, 7 May 2019 at 12:17, Francois E via ccache
 wrote:
> We use a shared ccache server hosted on a remote server, some of us
> experience slow connection to that server. I was wondering if
> compression would help in this case, i.e does the (un)compression occur
> at the server level or at the local level ?

Ccache does not have a server part, so compression is done locally. But I guess
you mean that you use a shared network filesystem, in which case compression
very well could help since the file server then won't have to receive and send
as much data.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Compression

[Francois E via ccache]

Hi,

We use a shared ccache server hosted on a remote server, some of us 
experience slow connection to that server. I was wondering if 
compression would help in this case, i.e does the (un)compression occur 
at the server level or at the local level ?


Thanks,
Francois

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression on or off by default?

[Anders Furuhed]

4 mar 2010 kl. 22.30 skrev Joel Rosdahl:

> On Wed, 3 Mar 2010 10:39:47 +
> "Dalton, Tom"  wrote:
> 
>> I would say for the first release of a new feature, default it to off. That
>> gives people a chance to 'play' with it without potentially breaking existing
>> installations that are simply upgrading. If there are no major problems with
>> the feature in that first release then make it a default in the next release
>> after that (e.g. ccache-3.1).
> 
> 
> My direct mode feature, on the other hand, is a much larger and less tested
> change, which I'm very well aware of. My secret plan was to try to get people
> to test my first announced release, and now the 3.0pre0 release, but neither
> release has triggered much reaction. So, maybe 3.0 will have to have direct
> mode disabled by default, unless I find a way to get more testers. :-)
> (Opinions are welcome, as usual.)
> 
> -- Joel

Joel, we've been using the 3.0pre0 release from day one, in a setup where a 
dirty dozen plus some droids share a cache.
All defaults, CCACHE_PREFIX=distcc.
I have nothing out of the ordinary to report. Thanks for your work!

Anders Furuhed
Pantor Engineering AB
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression on or off by default?

[Joel Rosdahl]

On Wed, 3 Mar 2010 10:39:47 +
"Dalton, Tom"  wrote:

> I would say for the first release of a new feature, default it to off. That
> gives people a chance to 'play' with it without potentially breaking existing
> installations that are simply upgrading. If there are no major problems with
> the feature in that first release then make it a default in the next release
> after that (e.g. ccache-3.1).

The compression feature is actually quite well-tested: it has been enabled by
default in Debian's ccache packages since 2007. But I'm leaning on disabling
compression by default for ccache anyway. Score so far: 4 votes (2 off-list)
for "default to off" and 0 votes for "default to on".

My direct mode feature, on the other hand, is a much larger and less tested
change, which I'm very well aware of. My secret plan was to try to get people
to test my first announced release, and now the 3.0pre0 release, but neither
release has triggered much reaction. So, maybe 3.0 will have to have direct
mode disabled by default, unless I find a way to get more testers. :-)
(Opinions are welcome, as usual.)

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression on or off by default?

[Mike Frysinger]

On Thu, Mar 4, 2010 at 16:00, Joel Rosdahl wrote:
> On Wed, 3 Mar 2010 05:00:48 -0500 Mike Frysinger wrote:
>> make the default a ./configure option that defaults to off
>
> I would prefer not to make a build-time option of this, because then the
> documentation has say something in line with "whether files are compressed by
> default depends on how ccache was built", which I think would be unfortunate. 
> I
> chose to not include the --{en,dis}able-zlib option from the original patch 
> for
> the same reason. I think it's better to strive for good defaults and few
> build-time options.

i didnt mean the configure option was in place of the runtime knobs

regardless, unless the compression is very low overhead, the default
should be off
-mike
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression on or off by default?

[Joel Rosdahl]

On Wed, 3 Mar 2010 05:00:48 -0500
Mike Frysinger  wrote:

> make the default a ./configure option that defaults to off

I would prefer not to make a build-time option of this, because then the
documentation has say something in line with "whether files are compressed by
default depends on how ccache was built", which I think would be unfortunate. I
chose to not include the --{en,dis}able-zlib option from the original patch for
the same reason. I think it's better to strive for good defaults and few
build-time options.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression on or off by default?

[Dalton, Tom]

I would say for the first release of a new feature, default it to off. That 
gives people a chance to 'play' with it without potentially breaking existing 
installations that are simply upgrading. If there are no major problems with 
the feature in that first release then make it a default in the next release 
after that (e.g. ccache-3.1).

-Original Message-
From: ccache-boun...@lists.samba.org [mailto:ccache-boun...@lists.samba.org] On 
Behalf Of Joel Rosdahl
Sent: 02 March 2010 22:11
To: ccache@lists.samba.org
Subject: [ccache] Compression on or off by default?

Hi,

Lars Gustäbel's compression patch (which will be incorporated in ccache 3.0) 
enables compression by default, and if you don't want compression you have to 
set CCACHE_NOCOMPRESS. I'm still a bit undecided about whether defaulting to 
compression is a good idea, though. Maybe we should be more conservative here 
and require CCACHE_COMPRESS to be set to enable compression instead? (Note that 
the question only is about the default behaviour when storing files in the 
cache -- ccache will still be able to read compressed and uncompressed files 
from the cache regardless of the CCACHE_(NO)COMPRESS setting.)

The main argument I see for making compression opt-in is that hard-linking 
doesn't work for compressed files (where "doesn't work" means that ccache will 
fall back to copying), so if you would like to try out hard-linking, you must 
set both CCACHE_NOCOMPRESS and CCACHE_HARDLINK, and also build up the cache 
again. Or, if you currently have enabled hard-linking with ccache 2.4, you need 
to take the explicit action of disabling compression after an upgrade to get 
the previous behaviour.

Another argument is maybe that disk space is cheap nowadays, and most people 
probably want to optimize for speed instead of disk space. On the other hand, 
the overhead of using compression is very small. In fact, I am unable to 
consistently measure any performance impact whatsoever. (Lars Gustäbel's own 
measurements can be found at <http://gustaebel.de/lars/ccache/>.) And, by 
compressing the cached files, more files will fit in the cache and also in the 
OS disk cache.

Does anyone have an opinion to share about this?

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression on or off by default?

[Mike Frysinger]

On Tue, Mar 2, 2010 at 17:10, Joel Rosdahl wrote:
> Lars Gustäbel's compression patch (which will be incorporated in ccache 3.0)
> enables compression by default, and if you don't want compression you have to
> set CCACHE_NOCOMPRESS. I'm still a bit undecided about whether defaulting to
> compression is a good idea, though. Maybe we should be more conservative here
> and require CCACHE_COMPRESS to be set to enable compression instead? (Note 
> that
> the question only is about the default behaviour when storing files in the
> cache -- ccache will still be able to read compressed and uncompressed files
> from the cache regardless of the CCACHE_(NO)COMPRESS setting.)
>
> The main argument I see for making compression opt-in is that hard-linking
> doesn't work for compressed files (where "doesn't work" means that ccache will
> fall back to copying), so if you would like to try out hard-linking, you must
> set both CCACHE_NOCOMPRESS and CCACHE_HARDLINK, and also build up the cache
> again. Or, if you currently have enabled hard-linking with ccache 2.4, you 
> need
> to take the explicit action of disabling compression after an upgrade to get
> the previous behaviour.
>
> Another argument is maybe that disk space is cheap nowadays, and most people
> probably want to optimize for speed instead of disk space. On the other hand,
> the overhead of using compression is very small. In fact, I am unable to
> consistently measure any performance impact whatsoever. (Lars Gustäbel's own
> measurements can be found at .) And, by
> compressing the cached files, more files will fit in the cache and also in the
> OS disk cache.
>
> Does anyone have an opinion to share about this?

make the default a ./configure option that defaults to off
-mike
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Compression on or off by default?

[Joel Rosdahl]

Hi,

Lars Gustäbel's compression patch (which will be incorporated in ccache 3.0)
enables compression by default, and if you don't want compression you have to
set CCACHE_NOCOMPRESS. I'm still a bit undecided about whether defaulting to
compression is a good idea, though. Maybe we should be more conservative here
and require CCACHE_COMPRESS to be set to enable compression instead? (Note that
the question only is about the default behaviour when storing files in the
cache -- ccache will still be able to read compressed and uncompressed files
from the cache regardless of the CCACHE_(NO)COMPRESS setting.)

The main argument I see for making compression opt-in is that hard-linking
doesn't work for compressed files (where "doesn't work" means that ccache will
fall back to copying), so if you would like to try out hard-linking, you must
set both CCACHE_NOCOMPRESS and CCACHE_HARDLINK, and also build up the cache
again. Or, if you currently have enabled hard-linking with ccache 2.4, you need
to take the explicit action of disabling compression after an upgrade to get
the previous behaviour.

Another argument is maybe that disk space is cheap nowadays, and most people
probably want to optimize for speed instead of disk space. On the other hand,
the overhead of using compression is very small. In fact, I am unable to
consistently measure any performance impact whatsoever. (Lars Gustäbel's own
measurements can be found at .) And, by
compressing the cached files, more files will fit in the cache and also in the
OS disk cache.

Does anyone have an opinion to share about this?

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache compression patch (was: The future of ccache)

[Joel Rosdahl]

On Thu, 7 Jan 2010 09:22:27 +1100
tridge at samba.org wrote:

> For example, the compression patch is badly
> done I think (it doesn't cope with changing the compression option -
> it should either use a .gz extension for compressed cache files, or
> recognise them by reading the header).

Hm, I don't see that problem with the patch. Can you elaborate?
Regardless of the CCACHE_NOCOMPRESS setting and the type of file, it
uses gzdopen+gzread to read the file, and gzread works on uncompressed
files too.

The only problem I saw with the patch is that it breaks forward
compatibility of ccache versions without the compression patch since
the hash sum and therefore the file name is the same for compressed and
uncompressed cache files. I think that's unfortunate, so to avoid this,
I made the hash sums for compressed and uncompressed files different.
A .gz extension for compressed files would of course work too, with
different trade-offs.

Regards,
Joel