Re: [ccache] ccache and precompiled headers

[Joel Rosdahl]

On 2010-08-27 15:59, Tor Arne Vestbø wrote:
> On 27.08.10 13.15, Tor Arne Vestbø wrote:
>> I'm sorry to report that I'm seeing build issues with master when
>> building Qt on Mac OS X (but the issue should be fixable):
> 
>> I'm assuming the hash is based solely on the contents of the
>> preprocessed output, not the flags/defines that were passed? But it
>> seems that the defines are stored somewhere in the precompiled header,
>> since it's barfing out when detecting a mismatch.
> 
> It seems we do include a few flags in the hash as well, but not -D.
> Using the following patch, with CCACHE_SLOPPINESS=time_macros set
> everything seems to work fine:

Great! Thanks for investigating this.

> I moved output_is_precompiled_header to be a global variable so it could
> be read in calculate_object_hash(), there's probably a nicer way to do
> that, passing it back from cc_process_args in an out-argument and
> then back into calculate_object_hash() or something similar.

Since there's already lots of global state, I think adding one more
variable doesn't make it worse. (I have plans to deglobalize state in
the future.)

> I also include all flags in the hash when output_is_precompiled_header
> is set, just to see if things would work. We could perhaps limit this to
> -D and a few others if we can figure out exactly what GCC puts into the
> precompiled header.

Right.

> Finally, I added an improved check for use of precompiled headers,
> checking for the existence of a .gch directory in -include arguments
> (instead of "pch" in the filename as before). You're right that we can't
> detect all uses, and modifying the build system is the sure-safe way,
> but I think we can at least try to be smart here?

I like that approach better, yes. I don't think we need to check that
pchpath is a directory, though, since the preprocessed header may be a
file as well.

I'll apply your patch and remove the directory check.

> With this setup I do get mostly direct hits on subsequent runs. The few
> preprocessed hits I get happens when looking for the object file hash in
> the manifest and not finding the manifest. Not sure why this would happen?

Then the hash of the arguments and the input file has changed. It's a
bit fiddly but possible to debug this. If you want to do it, you can
compile ccache with -DCCACHE_DEBUG_HASH and set CCACHE_DEBUG_HASH=1 in
the environment. ccache will then write the bytes it hashes to the file
ccache-debug-hash.bin so that you can compare the hashed bytes from two
different runs in a binary editor.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache and precompiled headers

[Tor Arne Vestbø]

On 27.08.10 13.15, Tor Arne Vestbø wrote:

On 22.08.10 12.42, Joel Rosdahl wrote:

On 2010-08-21 16:30, Joel Rosdahl wrote:

I've started implementing support for preprocessed headers (.gch)


Done.

Tor Arne: It would be great if you could try the current dev version of
ccache and see if it works well for you and if you get direct mode hits
when you use precompiled headers.


I'm sorry to report that I'm seeing build issues with master when
building Qt on Mac OS X (but the issue should be fixable):



I'm assuming the hash is based solely on the contents of the
preprocessed output, not the flags/defines that were passed? But it
seems that the defines are stored somewhere in the precompiled header,
since it's barfing out when detecting a mismatch.


It seems we do include a few flags in the hash as well, but not -D. 
Using the following patch, with CCACHE_SLOPPINESS=time_macros set 
everything seems to work fine:


diff --git i/ccache.c w/ccache.c
index d5b292d..45359ee 100644
--- i/ccache.c
+++ w/ccache.c
@@ -189,6 +189,9 @@ static int nlevels = 2;
  */
 static int compile_preprocessed_source_code;

+/* Whether the output is a precompiled header */
+static int output_is_precompiled_header = 0;
+
 /* How long (in microseconds) to wait before breaking a stale lock. */
 unsigned lock_staleness_limit = 200;

@@ -1019,8 +1022,8 @@ calculate_object_hash(struct args *args, struct 
mdfour *hash, int direct_mode)

/* When using the preprocessor, some arguments don't contribute
   to the hash. The theory is that these arguments will change
   the output of -E if they are going to have any effect at
-  all. */
-   if (!direct_mode) {
+  all. For precompiled headers this might not be the case. */
+   if (!direct_mode && !output_is_precompiled_header) {
if (i < args->argc-1) {
if (str_eq(args->argv[i], "-D") ||
str_eq(args->argv[i], "-I") ||
@@ -1342,7 +1345,7 @@ cc_process_args(struct args *orig_args, struct 
args **preprocessor_args,

int found_c_opt = 0;
int found_S_opt = 0;
int found_arch_opt = 0;
-   int output_is_precompiled_header = 0;
+   int found_pch = 0;
const char *explicit_language = NULL; /* As specified with -x. */
 	const char *file_language;/* As deduced from file 
extension. */

const char *actual_language;  /* Language to actually use. */
@@ -1582,6 +1585,7 @@ cc_process_args(struct args *orig_args, struct 
args **preprocessor_args,

};
int j;
char *relpath;
+   char *pchpath;
for (j = 0; opts[j]; j++) {
if (str_eq(argv[i], opts[j])) {
if (i == argc-1) {
@@ -1594,6 +1598,14 @@ cc_process_args(struct args *orig_args, struct 
args **preprocessor_args,

args_add(stripped_args, argv[i]);
relpath = 
make_relative_path(x_strdup(argv[i+1]));
args_add(stripped_args, relpath);
+
+   /* Try to be smart about detecting 
precompiled headers */
+   pchpath = format("%s.gch", argv[i+1]);
+   if (stat(pchpath, &st) == 0 && 
S_ISDIR(st.st_mode)) {
+   found_pch = 1;
+   }
+
+   free(pchpath);
free(relpath);
i++;
break;
@@ -1815,6 +1827,10 @@ cc_process_args(struct args *orig_args, struct 
args **preprocessor_args,

if (input_charset) {
args_add(*preprocessor_args, input_charset);
}
+   if (found_pch) {
+   cc_log("Use of precompiled header detected, adding 
-fpch-preprocess");
+   args_add(*preprocessor_args, "-fpch-preprocess");
+   }
if (explicit_language) {
args_add(*preprocessor_args, "-x");
args_add(*preprocessor_args, explicit_language);


I moved output_is_precompiled_header to be a global variable so it could 
be read in calculate_object_hash(), there's probably a nicer way to do 
that, passing it back from cc_process_args in an out-argument and

then back into calculate_object_hash() or something similar.

I also include all flags in the hash when output_is_precompiled_header 
is set, just to see if things would work. We could perhaps limit this to 
-D and a few others if we can figure out exactly what GCC puts into the 
precompiled header.


Finally, I added an improved check for use of precompiled h

Re: [ccache] ccache and precompiled headers

[Tor Arne Vestbø]

On 22.08.10 12.42, Joel Rosdahl wrote:

On 2010-08-21 16:30, Joel Rosdahl wrote:

I've started implementing support for preprocessed headers (.gch)


Done.

Tor Arne: It would be great if you could try the current dev version of
ccache and see if it works well for you and if you get direct mode hits
when you use precompiled headers.


I'm sorry to report that I'm seeing build issues with master when 
building Qt on Mac OS X (but the issue should be fixable):


[whopper:~/build/qt/4.7-cocoa-32/src/network] $  g++ -c -include 
.pch/debug-shared/QtNetwork -Winvalid-pch -pipe -Xarch_i386 
-mmacosx-version-min=10.4 -fconstant-cfstrings -O2 -arch i386 
-fvisibility=hidden -fvisibility-inlines-hidden -Wall -W -fPIC 
-DQT_SHARED -DQT_BUILD_NETWORK_LIB -DQT_NO_USING_NAMESPACE 
-DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT_MOC_COMPAT 
-DQT_USE_FAST_OPERATOR_PLUS -DQT_USE_FAST_CONCATENATION -DQT_NO_DEBUG 
-DQT_CORE_LIB -DQT_HAVE_MMX -DQT_HAVE_3DNOW -DQT_HAVE_SSE 
-DQT_HAVE_MMXEXT -DQT_HAVE_SSE2 -DQT_HAVE_SSE3 -DQT_HAVE_SSSE3 
-DQT_HAVE_SSE4_1 -DQT_HAVE_SSE4_2 -D_LARGEFILE64_SOURCE 
-D_LARGEFILE_SOURCE -I/Users/torarne/dev/qt/4.7/mkspecs/macx-g++ 
-I/Users/torarne/dev/qt/4.7/src/network -I../../include/QtCore 
-I../../include -I../../include/QtNetwork -I.rcc/debug-shared 
-I/Users/torarne/dev/qt/4.7/src/network/kernel -I.moc/release-shared -I. 
-F/Users/torarne/build/qt/4.7-cocoa-32/lib -o .obj/release-shared/qftp.o 
/Users/torarne/dev/qt/4.7/src/network/access/qftp.cpp
cc1plus: warning: ./.pch/debug-shared/QtNetwork.gch/c++: not used 
because `QT_BUILD_XML_LIB' not defined
cc1plus: warning: ./.pch/debug-shared/QtNetwork.gch/objective-c++: not 
for GNU C++
cc1plus: warning: ./.pch/debug-shared/QtNetwork.gch/c++: not used 
because `QT_BUILD_XML_LIB' not defined
cc1plus: warning: ./.pch/debug-shared/QtNetwork.gch/objective-c++: not 
for GNU C++

cc1plus: error: .pch/debug-shared/QtNetwork: No such file or directory
cc1plus: error: one or more PCH files were found, but they were invalid

The reason seems to be that the object file hashing fails, and we end up 
using a cached version of the precompiled header from an earlier step in 
the compilation, where another define was set.


Here's when it's created, for the QtXml module:

[08-26T17:59:05.644922 70725] === CCACHE STARTED 
=

[2010-08-26T17:59:05.645521 70725] Hostname: whopper
[2010-08-26T17:59:05.645533 70725] Working directory: 
/Users/torarne/build/qt/4.7-cocoa-32/src/xml
[2010-08-26T17:59:05.645566 70725] i386 is not a regular file, not 
considering as input file
[2010-08-26T17:59:05.645622 70725] Creating precompiled header; not 
compiling preprocessed code
[2010-08-26T17:59:05.645666 70725] Source file: 
/Users/torarne/dev/qt/4.7/src/corelib/global/qt_pch.h
[2010-08-26T17:59:05.645674 70725] Object file: 
.pch/debug-shared/QtXml.gch/c++

[2010-08-26T17:59:05.645692 70725] Trying direct lookup
[2010-08-26T17:59:05.645762 70725] Looking for object file hash in 
/Users/torarne/.ccache/d/8/a4db5d4c8aa3c1e487b474b990bc72-1807.manifest

[2010-08-26T17:59:05.645780 70725] No such manifest file
[2010-08-26T17:59:05.645787 70725] Did not find object file hash in manifest
[2010-08-26T17:59:05.645793 70725] Running preprocessor
[2010-08-26T17:59:05.645810 70725] Executing /usr/bin/g++ -pipe 
-Xarch_i386 -mmacosx-version-min=10.4 -fconstant-cfstrings -O2 -arch 
i386 -fvisibility=hidden -fvisibility-inlines-hidden -Wall -W -fPIC 
-DQT_SHARED -DQT_BUILD_XML_LIB -DQT_NO_USING_NAMESPACE 
-DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT_MOC_COMPAT 
-DQT_USE_FAST_OPERATOR_PLUS -DQT_USE_FAST_CONCATENATION -DQT_NO_DEBUG 
-DQT_CORE_LIB -DQT_HAVE_MMX -DQT_HAVE_3DNOW -DQT_HAVE_SSE 
-DQT_HAVE_MMXEXT -DQT_HAVE_SSE2 -DQT_HAVE_SSE3 -DQT_HAVE_SSSE3 
-DQT_HAVE_SSE4_1 -DQT_HAVE_SSE4_2 -D_LARGEFILE64_SOURCE 
-D_LARGEFILE_SOURCE -I/Users/torarne/dev/qt/4.7/mkspecs/macx-g++ 
-I/Users/torarne/dev/qt/4.7/src/xml -I../../include/QtCore 
-I../../include -I../../include/QtXml -I.rcc/debug-shared 
-I.moc/release-shared -I. -F/Users/torarne/build/qt/4.7-cocoa-32/lib -c 
-x c++-header -E /Users/torarne/dev/qt/4.7/src/corelib/global/qt_pch.h

[2010-08-26T17:59:05.737271 70725] Got object file hash from preprocessor
[2010-08-26T17:59:05.737365 70725] Object file 
/Users/torarne/.ccache/6/c/7236596f54a9fa1cadf7181f4ebfda-1090377.o not 
in cache

[2010-08-26T17:59:05.737381 70725] Running real compiler
[2010-08-26T17:59:05.737386 70725] Executing /usr/bin/g++ -pipe 
-Xarch_i386 -mmacosx-version-min=10.4 -fconstant-cfstrings -O2 -arch 
i386 -fvisibility=hidden -fvisibility-inlines-hidden -Wall -W -fPIC 
-DQT_SHARED -DQT_BUILD_XML_LIB -DQT_NO_USING_NAMESPACE 
-DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT_MOC_COMPAT 
-DQT_USE_FAST_OPERATOR_PLUS -DQT_USE_FAST_CONCATENATION -DQT_NO_DEBUG 
-DQT_CORE_LIB -DQT_HAVE_MMX -DQT_HAVE_3DNOW -DQT_HAVE_SSE 
-DQT_HAVE_MMXEXT -DQT_HAVE_SSE2 -DQT_HAVE_SSE3 -DQT_HAVE_SSSE3 
-DQT_HAVE_SSE4_1 -DQT_HAVE_SSE4_2 -D_LARGEFI

Re: [ccache] ccache and precompiled headers

[Tor Arne Vestbø]

On 22.08.10 12.42, Joel Rosdahl wrote:

On 2010-08-21 16:30, Joel Rosdahl wrote:

I've started implementing support for preprocessed headers (.gch)


Done.

Tor Arne: It would be great if you could try the current dev version of
ccache and see if it works well for you and if you get direct mode hits
when you use precompiled headers.


Cool! Will do and get back to you.

Tor Arne
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache and precompiled headers

[Joel Rosdahl]

On 2010-08-21 16:30, Joel Rosdahl wrote:
> I've started implementing support for preprocessed headers (.gch)

Done.

Tor Arne: It would be great if you could try the current dev version of
ccache and see if it works well for you and if you get direct mode hits
when you use precompiled headers.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache and precompiled headers

[Joel Rosdahl]

On 2010-07-03 14:22, Tor Arne Vestbø wrote:
> [...]
> As you can see removing pch from the build increases the total build
> time for the initial build (empty cache), but speeds up builds when
> there's something in the cache since we're hitting direct mode in
> almost all cases. When pch and ccache is used together the initial
> build time decreases, _but_ builds with a filled cache now take
> longer.
> 
> Seems like we're hitting a lot more preprocessed cases for some
> reason. Dunno why that might be?

I've started implementing support for preprocessed headers (.gch) and
now I know why you get lots of preprocessed hits with your suggested
patch. The thing is that a .gch file created by GCC is apparently not
"stable"; the content is different each time for the same input. And
since ccache currently doesn't know how to cache the creation of a
precompiled header, a new and thus different .gch file is created, so
the direct mode will therefore miss.

Another issue is that the direct mode hashing of source code skips over
comments. This means that if there happens to be a "/*" or "//" in the
.gch (which is a binary file), parts of the file will not contribute to
the hash sum, which is bad.

Both issues should be easy to handle, though.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache and precompiled headers

[Joel Rosdahl]

On 2010-07-03 14:22, Tor Arne Vestbø wrote:
> [...]
> Right, that would be a solution. I assume GCC does not barf it passed
> -fpch-preprocess in non-preprocess mode.

Yes, it works fine.

> [...]
> If that kind of compiler-feature-inspection is on the roadmap anyways
> that would indeed be an option, but I'm not sure it makes sense alone
> for this particular usecase since there is another workaround.

Yes, I think that we need to do compiler feature detection eventually.
The bug , which showed
up just a week ago, is a good example of a case where compiler-specific
behaviour would nice.

> [...]
> Hmm, this one worries me a bit. How are we going to figure out if the
> precompiled header contains __TIME__?

I don't think we can figure it out in a reasonable way, which is why we
need to always assume that __TIME__ is used if a preprocessed header is
used.

> [...]
> As you can see removing pch from the build increases the total build
> time for the initial build (empty cache), but speeds up builds when
> there's something in the cache since we're hitting direct mode in
> almost all cases. When pch and ccache is used together the initial
> build time decreases, _but_ builds with a filled cache now take
> longer.
>
> Seems like we're hitting a lot more preprocessed cases for some
> reason. Dunno why that might be?

Please run with CCACHE_LOGFILE and check if the log says why direct mode
gets disabled.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache and precompiled headers

[Tor Arne Vestbø]

Hey!

On Sun, Jun 27, 2010 at 2:15 PM, Joel Rosdahl  wrote:
> Hi,
>
> On 2010-06-21 13:09, Tor Arne Vestbř wrote:
>> I ran into a problem where ccache would log "Preprocessor gave exit
>> status 1" and fail to cache builds.
>>
>> The solution seems to be to add the -fpch-preprocess flag, which will
>> add a custom #pragma include to the preprocessed output,
>
> I came to mostly the same conclusion some months ago but decided to
> postpone thinking about support for preprocessed headers until after the
> 3.0 release. So, now's a good time to continue. :-)

Makes sense. Love the speed of the direct mode btw :)

> See comments below.
>
>> [...]
>> @@ -1557,6 +1562,12 @@ static void process_args(int argc, char **argv, ARGS 
>> **preprocessor_args,
>> }
>>
>> args_add(stripped_args, argv[i]);
>> +   if (strcmp(argv[i], "-include")
>> +   && strstr(argv[i+1], 
>> "pch")) {
>> +   found_pch = 1;
>> +   cc_log("Found precompiled 
>> header: %s",
>> +  argv[i+1]);
>> +   }
>
> But this means that you have to include "pch" in name of the precompiled
> header file, which is quite limiting. There's also the case when using
> '#include "preprocessed.h"' in the source code to include the
> precompiled header instead of using the -include compiler option. Also,
> if the name contains "pch" when it's not a precompiled header,
> -fpch-preprocess will be added anyway, and that will make the
> preprocessor fail if the compiler isn't GCC 4.0 or higher.

Yupp, that particular part of the patch was just to try things out for
my usecase where the files had "pch" in them. (also there's a bug in
the strcmp, which should do == 0)

> I don't see how ccache in a good way could figure out if the
> -fpch-preprocess option should be added or not, so the easy option is
> this: Just document that if you want ccache caching to work with
> precompiled headers, you have make sure to add the -fpch-preprocess
> option to the compilation command. That may be done by altering the
> build system (adding -fpch-preprocess to CPPFLAGS or similar), but it
> could also be done by adding -fpch-preprocess to CCACHE_CPPFLAGS if we
> introduce such a variable. We also need the first part of your patch, of
> course.

Right, that would be a solution. I assume GCC does not barf it passed
-fpch-preprocess in non-preprocess mode.

> Another way would be to ask the compiler if it supports the
> -fpch-preprocess option and, if so, always add it. This could be done by
> checking the status code of running something like "compiler
> -fpch-preprocess -x c -c /dev/null -o /dev/null". The result of this
> command could be cached somewhere in CCACHE_DIR (keyed with the
> compiler's mtime) so that there's no need to run it more than once.
> (Storing other types of information about the compiler may be useful for
> other things as well, like doing compiler-specific handling of flags or
> maybe enabling direct mode only for compilers known to work with it.)

If that kind of compiler-feature-inspection is on the roadmap anyways
that would indeed be an option, but I'm not sure it makes sense alone
for this particular usecase since there is another workaround.

> There's also at least one more subtle thing that needs to be addressed:
> the __TIME__ macro in direct mode. As can be read in the Performance
> section in the manual
> (http://ccache.samba.org/manual.html#_performance), the direct mode is
> turned off when "__TIME__" is present in the source code. But when
> precompiled headers are being used, ccache can't easily read the code
> that was included, so it can't tell whether __TIME__ is potentially
> being used and thus has to turn off direct mode as a safety measure
> (unless CCACHE_SLOPPY=time_macros). This also needs to be implemented.

Hmm, this one worries me a bit. How are we going to figure out if the
precompiled header contains __TIME__?

> What do you think?

I think if we can make it correct, supporting PCH would be a nice
addition. On the other hand, I did some quick build-tests with the
patch and here's what I found:

(numbers in parenthesis are for retries of the same config)


 vanlilla (with pch) . 10:33

- pch .. 12:42 (12:43)
+ ccache ... 13:18 (13:17)
 ccache 2nd clean rebuild ...  4:32 (3:53, 4:14, 4:10)

[whopper:~/dev/ccache] $ ccache -s
cache directory /Users/torarne/.ccache
cache hit (direct)  7655
cache hit (preprocessed)  23
cache miss  3816
called for link  219
unsupported source language6
files in cache  7720
cache size   

Re: [ccache] ccache and precompiled headers

[Joel Rosdahl]

Hi,

On 2010-06-21 13:09, Tor Arne Vestbø wrote:
> I ran into a problem where ccache would log "Preprocessor gave exit
> status 1" and fail to cache builds. Closer inspection revealed that
> this was due to the use of precompiled headers though an -include
> argument to gcc. When ccache then tried to run the preprocessor on the
> source file gcc reported "file not found" for the precompiled header.
> Aparenly gcc appends ".gch" and some other magic to the filename
> during build but not during preprocessing.
>
> The solution seems to be to add the -fpch-preprocess flag, which will
> add a custom #pragma include to the preprocessed output, see:
> 
>   
> http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gcc/Preprocessor-Options.html#index-fpch_002dpreprocess-877

I came to mostly the same conclusion some months ago but decided to
postpone thinking about support for preprocessed headers until after the
3.0 release. So, now's a good time to continue. :-)

> The question is, is it safe to use ccache with precompiled headers at all?
> 
> If so, would a patch like this work, basically adding the flag if a
> pch include is detected, and making sure it's included and hashed in
> the direct mode manifest's list of includes?

See comments below.

> [...]
> @@ -1557,6 +1562,12 @@ static void process_args(int argc, char **argv, ARGS 
> **preprocessor_args,
> }
> 
> args_add(stripped_args, argv[i]);
> +   if (strcmp(argv[i], "-include")
> +   && strstr(argv[i+1], 
> "pch")) {
> +   found_pch = 1;
> +   cc_log("Found precompiled 
> header: %s",
> +  argv[i+1]);
> +   }

But this means that you have to include "pch" in name of the precompiled
header file, which is quite limiting. There's also the case when using
'#include "preprocessed.h"' in the source code to include the
precompiled header instead of using the -include compiler option. Also,
if the name contains "pch" when it's not a precompiled header,
-fpch-preprocess will be added anyway, and that will make the
preprocessor fail if the compiler isn't GCC 4.0 or higher.

I don't see how ccache in a good way could figure out if the
-fpch-preprocess option should be added or not, so the easy option is
this: Just document that if you want ccache caching to work with
precompiled headers, you have make sure to add the -fpch-preprocess
option to the compilation command. That may be done by altering the
build system (adding -fpch-preprocess to CPPFLAGS or similar), but it
could also be done by adding -fpch-preprocess to CCACHE_CPPFLAGS if we
introduce such a variable. We also need the first part of your patch, of
course.

Another way would be to ask the compiler if it supports the
-fpch-preprocess option and, if so, always add it. This could be done by
checking the status code of running something like "compiler
-fpch-preprocess -x c -c /dev/null -o /dev/null". The result of this
command could be cached somewhere in CCACHE_DIR (keyed with the
compiler's mtime) so that there's no need to run it more than once.
(Storing other types of information about the compiler may be useful for
other things as well, like doing compiler-specific handling of flags or
maybe enabling direct mode only for compilers known to work with it.)

There's also at least one more subtle thing that needs to be addressed:
the __TIME__ macro in direct mode. As can be read in the Performance
section in the manual
(http://ccache.samba.org/manual.html#_performance), the direct mode is
turned off when "__TIME__" is present in the source code. But when
precompiled headers are being used, ccache can't easily read the code
that was included, so it can't tell whether __TIME__ is potentially
being used and thus has to turn off direct mode as a safety measure
(unless CCACHE_SLOPPY=time_macros). This also needs to be implemented.

What do you think?

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache and precompiled headers

[Tor Arne Vestbø]

Hey,

I ran into a problem where ccache would log "Preprocessor gave exit
status 1" and fail to cache builds. Closer inspection revealed that
this was due to the use of precompiled headers though an -include
argument to gcc. When ccache then tried to run the preprocessor on the
source file gcc reported "file not found" for the precompiled header.
Aparenly gcc appends ".gch" and some other magic to the filename
during build but not during preprocessing.

The solution seems to be to add the -fpch-preprocess flag, which will
add a custom #pragma include to the preprocessed output, see:

  
http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gcc/Preprocessor-Options.html#index-fpch_002dpreprocess-877

The question is, is it safe to use ccache with precompiled headers at all?

If so, would a patch like this work, basically adding the flag if a
pch include is detected, and making sure it's included and hashed in
the direct mode manifest's list of includes?

diff --git i/ccache.c w/ccache.c
index 310d3e2..40d0e9d 100644
--- i/ccache.c
+++ w/ccache.c
@@ -477,6 +477,7 @@ static int process_preprocessed_file(struct mdfour
*hash, const char *path)
 *
 *   # N "file"
 *   # N "file" N
+*   #pragma GCC pch_preprocess "file"
 *
 * HP's compiler:
 *
@@ -488,6 +489,9 @@ static int process_preprocessed_file(struct mdfour
*hash, const char *path)
if (q[0] == '#'
/* GCC: */
&& ((q[1] == ' ' && q[2] >= '0' && q[2] <= '9')
+|| (q[1] == 'p' && q[2] == 'r' && q[3] == 'a'
&& q[4] == 'g'
+&& q[5] == 'm' && q[6] == 'a' && (q < end - 30)
+&& (strncmp(q + 8, "GCC pch_preprocess", 18) == 0))
/* HP: */
|| (q[1] == 'l' && q[2] == 'i' && q[3] == 'n'
&& q[4] == 'e'
&& q[5] == ' '))
@@ -1336,6 +1340,7 @@ static void process_args(int argc, char **argv,
ARGS **preprocessor_args,
int found_c_opt = 0;
int found_S_opt = 0;
int found_arch_opt = 0;
+   int found_pch = 0;
const char *explicit_language = NULL; /* As specified with -x. */
const char *file_language;/* As deduced from file
extension. */
const char *actual_language;  /* Language to actually use. */
@@ -1557,6 +1562,12 @@ static void process_args(int argc, char **argv,
ARGS **preprocessor_args,
}

args_add(stripped_args, argv[i]);
+   if (strcmp(argv[i], "-include")
+   &&
strstr(argv[i+1], "pch")) {
+   found_pch = 1;
+   cc_log("Found
precompiled header: %s",
+  argv[i+1]);
+   }
relpath =
make_relative_path(x_strdup(argv[i+1]));
args_add(stripped_args, relpath);
free(relpath);
@@ -1781,6 +1792,9 @@ static void process_args(int argc, char **argv,
ARGS **preprocessor_args,
if (input_charset) {
args_add(*preprocessor_args, input_charset);
}
+   if (found_pch) {
+   args_add(*preprocessor_args, "-fpch-preprocess");
+   }
if (explicit_language) {
args_add(*preprocessor_args, "-x");
args_add(*preprocessor_args, explicit_language);

Thanks for any input!

Tor Arne
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache