Re: [PATCH] offline gcda profile processing tool

2014-07-13 Thread Andreas Schwab
Rong Xu x...@google.com writes:

 In my test, I used gcc-4.6 as the host compiler. I got warning like this:

 In file included from ../../gcc/gcc/../libgcc/libgcov-util.c:30:0:
 ../../gcc/gcc/../libgcc/libgcov.h:184:30: warning: ISO C++ forbids
 zero-size array ‘ctrs’ [-pedantic]

With gcc 4.3 this is an error, so -Wno-error has no effect.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
And now for something completely different.


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Richard Biener
On Mon, May 5, 2014 at 7:17 PM, Rong Xu x...@google.com wrote:
 Here is the updated patch. The difference with patch set 3 is
 (1) use the gcov-counter.def for scaling operation.
 (2) fix wrong scaling function for time-profiler.

 passed with bootstrap, profiledboostrap and SPEC2006.

One of the patches breaks bootstrap for me:

/space/rguenther/src/svn/trunk/gcc/../libgcc/libgcov.h:184: error: ISO
C++ forbids zero-size array ‘ctrs’
make[3]: *** [libgcov-util.o] Error 1

host compiler is gcc 4.3.4

Richard.

 Thanks,

 -Rong

 On Thu, May 1, 2014 at 3:37 PM, Rong Xu x...@google.com wrote:
 Hi, Honza,

 Please find the new patch in the attachment. This patch integrated
 your earlier suggestions. The noticeable changes are:
 (1) build specialized object for libgcov-driver.c and libgcov-merge.c
 and link into gcov-tool, rather including the source file.
 (2) split some gcov counter definition code to gcov-counter.def to
 avoid code duplication.
 (3) use a macro for gcov_read_counter(), so gcov-tool can use existing
 code in libgcov-merge.c with minimal change.

 This patch does not address the suggestion of creating a new directory
 for libgcov. I agree with you that's a much better
 and cleaner structure we should go for. We can do that in follow-up patches.

 I tested this patch with boostrap and profiledbootstrap. Other tests
 are on-going.

 Thanks,

 -Rong

 On Wed, Apr 16, 2014 at 8:34 PM, Jan Hubicka hubi...@ucw.cz wrote:
 GCOT_TOOL needs to use this function to read the string in gcda file
 to memory to construct gcov_info objects.
 As you noticed, gcov runtime does not need this interface. But
 gcov-tool links with gcov runtime and it also uses the function.
 We could make it available in gcov_runtime, but that will slightly
 increase the memory footprint.

 Yep, it is not really pretty. I wrote bellow some plan how things may be
 structured in more convenient way.  Any work in that direction would be 
 welcome.

 The planned new functions for trunk version is: (1) overlap score b/w
 two profiles (2) better dumping (top hot objects/function/counters)
 and statistics.
 Once this basic version is in, we can start to add the new functionality.

 Sounds good. I assume the autoFDO does not go via gcov tool but rather uses
 custom reader of profile data at GCC side?
 I wonder, are there any recent overview papers/slides/text of design of 
 AutoFDO
 and other features to be merged?
 I remember the talk from GCC Summit and I did read some of pre-LTO LIPO
 sources  papers, but it would be nice to have somethin up to date.

 libgcov-util.o is built in gcc/ directory, rather in libgcc.
 It's directly linked to gcov-tool.
 So libgcov-util.o is built for HOST, not TARGET.
 With makefile changes, we can built HOST version of libgcov-driver.o
 and libgcov-merge.o.
 I also need to make some static functions/variables public.

 I suppose that can go with IN_GCOV_TOOL ifdef.

 So we currently have basic gcov io handling in gcc/gcov-io.c that is 
 borrowed
 by libgcc/libgcov* code.  We also will get libgcov-util.c in libgcc 
 directory
 that is actually borrowed by by gcc/gcov-tool.c code.

 We now have one runtime using STDIO for streaming and kernel has custom 
 version
 that goes via /proc interface (last time I looked).  We added some 
 abstraction
 into libgcov-interface that is the part that relies on STDIO, partly via 
 gcov-io.c
 code and now we have in-memory handling code in libgcov-util.

 I guess it would make most sentse to put all the gcov code into a new 
 directory
 (libgcov) and make it stand-alone library that can be configured
 1) for stdio based runtime as we do not
 2) for runtime missing the interface and relyin on user providing it
 3) for use within gcov file manipulation tools with reorg of
 GCC/gcov/gcov-dump/gcov-tool to all use the same low-level interfaces.
 In a longer term, I would like to make FDO profiling intstrumentation to 
 happen
 at linktime. For that I need to make the instrumentation code (minimal 
 spaning
 tree  friends) to work w/o cgraph that would ideally be done in a shared
 implementation
  Won't this get wrong answer when counters[0] is not the same?
  I would expected the merging code to compare the counters first. 
  Similarly for delta counter.

 These *_op functions are for scaling only. So there is only one
 profile involved (thus there is no comparison).
 The merge handles are in libgcov-merge.c which have the code to handle
 mismatched profile targets.

 I see, OK then.
 
  Adding correct rounding may actually make difference for Martin's 
  startup time work.

 Do you mean to use something like in RDIV macro?

 Yes.

 Honza


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Christophe Lyon
On 11 July 2014 10:07, Richard Biener richard.guent...@gmail.com wrote:
 On Mon, May 5, 2014 at 7:17 PM, Rong Xu x...@google.com wrote:
 Here is the updated patch. The difference with patch set 3 is
 (1) use the gcov-counter.def for scaling operation.
 (2) fix wrong scaling function for time-profiler.

 passed with bootstrap, profiledboostrap and SPEC2006.

 One of the patches breaks bootstrap for me:

 /space/rguenther/src/svn/trunk/gcc/../libgcc/libgcov.h:184: error: ISO
 C++ forbids zero-size array ‘ctrs’
 make[3]: *** [libgcov-util.o] Error 1

 host compiler is gcc 4.3.4

 Richard.


On my side, commit 212448 breaks the cross-build of GCC for targets
using newlib:
* arm-none-eabi
* aarch64-none-elf
/usr/include/sys/stat.h: In function 8098void
gcov_output_files(const char*, gcov_info*)8099:
/usr/include/sys/stat.h:317: error: too few arguments to function
8098int mkdir(const char*, __mode_t)8099
/tmp/1392958_1.tmpdir/aci-gcc-fsf/sources/gcc-fsf/gccsrc/gcc/gcov-tool.c:96:
error: at this point in file
make[2]: *** [gcov-tool.o] Error 1

Christophe.

 Thanks,

 -Rong

 On Thu, May 1, 2014 at 3:37 PM, Rong Xu x...@google.com wrote:
 Hi, Honza,

 Please find the new patch in the attachment. This patch integrated
 your earlier suggestions. The noticeable changes are:
 (1) build specialized object for libgcov-driver.c and libgcov-merge.c
 and link into gcov-tool, rather including the source file.
 (2) split some gcov counter definition code to gcov-counter.def to
 avoid code duplication.
 (3) use a macro for gcov_read_counter(), so gcov-tool can use existing
 code in libgcov-merge.c with minimal change.

 This patch does not address the suggestion of creating a new directory
 for libgcov. I agree with you that's a much better
 and cleaner structure we should go for. We can do that in follow-up patches.

 I tested this patch with boostrap and profiledbootstrap. Other tests
 are on-going.

 Thanks,

 -Rong

 On Wed, Apr 16, 2014 at 8:34 PM, Jan Hubicka hubi...@ucw.cz wrote:
 GCOT_TOOL needs to use this function to read the string in gcda file
 to memory to construct gcov_info objects.
 As you noticed, gcov runtime does not need this interface. But
 gcov-tool links with gcov runtime and it also uses the function.
 We could make it available in gcov_runtime, but that will slightly
 increase the memory footprint.

 Yep, it is not really pretty. I wrote bellow some plan how things may be
 structured in more convenient way.  Any work in that direction would be 
 welcome.

 The planned new functions for trunk version is: (1) overlap score b/w
 two profiles (2) better dumping (top hot objects/function/counters)
 and statistics.
 Once this basic version is in, we can start to add the new functionality.

 Sounds good. I assume the autoFDO does not go via gcov tool but rather uses
 custom reader of profile data at GCC side?
 I wonder, are there any recent overview papers/slides/text of design of 
 AutoFDO
 and other features to be merged?
 I remember the talk from GCC Summit and I did read some of pre-LTO LIPO
 sources  papers, but it would be nice to have somethin up to date.

 libgcov-util.o is built in gcc/ directory, rather in libgcc.
 It's directly linked to gcov-tool.
 So libgcov-util.o is built for HOST, not TARGET.
 With makefile changes, we can built HOST version of libgcov-driver.o
 and libgcov-merge.o.
 I also need to make some static functions/variables public.

 I suppose that can go with IN_GCOV_TOOL ifdef.

 So we currently have basic gcov io handling in gcc/gcov-io.c that is 
 borrowed
 by libgcc/libgcov* code.  We also will get libgcov-util.c in libgcc 
 directory
 that is actually borrowed by by gcc/gcov-tool.c code.

 We now have one runtime using STDIO for streaming and kernel has custom 
 version
 that goes via /proc interface (last time I looked).  We added some 
 abstraction
 into libgcov-interface that is the part that relies on STDIO, partly via 
 gcov-io.c
 code and now we have in-memory handling code in libgcov-util.

 I guess it would make most sentse to put all the gcov code into a new 
 directory
 (libgcov) and make it stand-alone library that can be configured
 1) for stdio based runtime as we do not
 2) for runtime missing the interface and relyin on user providing it
 3) for use within gcov file manipulation tools with reorg of
 GCC/gcov/gcov-dump/gcov-tool to all use the same low-level interfaces.
 In a longer term, I would like to make FDO profiling intstrumentation to 
 happen
 at linktime. For that I need to make the instrumentation code (minimal 
 spaning
 tree  friends) to work w/o cgraph that would ideally be done in a shared
 implementation
  Won't this get wrong answer when counters[0] is not the same?
  I would expected the merging code to compare the counters first. 
  Similarly for delta counter.

 These *_op functions are for scaling only. So there is only one
 profile involved (thus there is no comparison).
 The merge handles are in libgcov-merge.c which have the code to 

Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Xinliang David Li
I wonder why. The struct definition for gcov_fn_info has not changed
in this patch.

David

On Fri, Jul 11, 2014 at 1:07 AM, Richard Biener
richard.guent...@gmail.com wrote:
 On Mon, May 5, 2014 at 7:17 PM, Rong Xu x...@google.com wrote:
 Here is the updated patch. The difference with patch set 3 is
 (1) use the gcov-counter.def for scaling operation.
 (2) fix wrong scaling function for time-profiler.

 passed with bootstrap, profiledboostrap and SPEC2006.

 One of the patches breaks bootstrap for me:

 /space/rguenther/src/svn/trunk/gcc/../libgcc/libgcov.h:184: error: ISO
 C++ forbids zero-size array ‘ctrs’
 make[3]: *** [libgcov-util.o] Error 1

 host compiler is gcc 4.3.4

 Richard.

 Thanks,

 -Rong

 On Thu, May 1, 2014 at 3:37 PM, Rong Xu x...@google.com wrote:
 Hi, Honza,

 Please find the new patch in the attachment. This patch integrated
 your earlier suggestions. The noticeable changes are:
 (1) build specialized object for libgcov-driver.c and libgcov-merge.c
 and link into gcov-tool, rather including the source file.
 (2) split some gcov counter definition code to gcov-counter.def to
 avoid code duplication.
 (3) use a macro for gcov_read_counter(), so gcov-tool can use existing
 code in libgcov-merge.c with minimal change.

 This patch does not address the suggestion of creating a new directory
 for libgcov. I agree with you that's a much better
 and cleaner structure we should go for. We can do that in follow-up patches.

 I tested this patch with boostrap and profiledbootstrap. Other tests
 are on-going.

 Thanks,

 -Rong

 On Wed, Apr 16, 2014 at 8:34 PM, Jan Hubicka hubi...@ucw.cz wrote:
 GCOT_TOOL needs to use this function to read the string in gcda file
 to memory to construct gcov_info objects.
 As you noticed, gcov runtime does not need this interface. But
 gcov-tool links with gcov runtime and it also uses the function.
 We could make it available in gcov_runtime, but that will slightly
 increase the memory footprint.

 Yep, it is not really pretty. I wrote bellow some plan how things may be
 structured in more convenient way.  Any work in that direction would be 
 welcome.

 The planned new functions for trunk version is: (1) overlap score b/w
 two profiles (2) better dumping (top hot objects/function/counters)
 and statistics.
 Once this basic version is in, we can start to add the new functionality.

 Sounds good. I assume the autoFDO does not go via gcov tool but rather uses
 custom reader of profile data at GCC side?
 I wonder, are there any recent overview papers/slides/text of design of 
 AutoFDO
 and other features to be merged?
 I remember the talk from GCC Summit and I did read some of pre-LTO LIPO
 sources  papers, but it would be nice to have somethin up to date.

 libgcov-util.o is built in gcc/ directory, rather in libgcc.
 It's directly linked to gcov-tool.
 So libgcov-util.o is built for HOST, not TARGET.
 With makefile changes, we can built HOST version of libgcov-driver.o
 and libgcov-merge.o.
 I also need to make some static functions/variables public.

 I suppose that can go with IN_GCOV_TOOL ifdef.

 So we currently have basic gcov io handling in gcc/gcov-io.c that is 
 borrowed
 by libgcc/libgcov* code.  We also will get libgcov-util.c in libgcc 
 directory
 that is actually borrowed by by gcc/gcov-tool.c code.

 We now have one runtime using STDIO for streaming and kernel has custom 
 version
 that goes via /proc interface (last time I looked).  We added some 
 abstraction
 into libgcov-interface that is the part that relies on STDIO, partly via 
 gcov-io.c
 code and now we have in-memory handling code in libgcov-util.

 I guess it would make most sentse to put all the gcov code into a new 
 directory
 (libgcov) and make it stand-alone library that can be configured
 1) for stdio based runtime as we do not
 2) for runtime missing the interface and relyin on user providing it
 3) for use within gcov file manipulation tools with reorg of
 GCC/gcov/gcov-dump/gcov-tool to all use the same low-level interfaces.
 In a longer term, I would like to make FDO profiling intstrumentation to 
 happen
 at linktime. For that I need to make the instrumentation code (minimal 
 spaning
 tree  friends) to work w/o cgraph that would ideally be done in a shared
 implementation
  Won't this get wrong answer when counters[0] is not the same?
  I would expected the merging code to compare the counters first. 
  Similarly for delta counter.

 These *_op functions are for scaling only. So there is only one
 profile involved (thus there is no comparison).
 The merge handles are in libgcov-merge.c which have the code to handle
 mismatched profile targets.

 I see, OK then.
 
  Adding correct rounding may actually make difference for Martin's 
  startup time work.

 Do you mean to use something like in RDIV macro?

 Yes.

 Honza


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Xinliang David Li
What is the macro to test POSIX IO on host? The guard uses
TARGET_POSIX_IO which is not right.

David

On Fri, Jul 11, 2014 at 1:12 AM, Christophe Lyon
christophe.l...@linaro.org wrote:
 On 11 July 2014 10:07, Richard Biener richard.guent...@gmail.com wrote:
 On Mon, May 5, 2014 at 7:17 PM, Rong Xu x...@google.com wrote:
 Here is the updated patch. The difference with patch set 3 is
 (1) use the gcov-counter.def for scaling operation.
 (2) fix wrong scaling function for time-profiler.

 passed with bootstrap, profiledboostrap and SPEC2006.

 One of the patches breaks bootstrap for me:

 /space/rguenther/src/svn/trunk/gcc/../libgcc/libgcov.h:184: error: ISO
 C++ forbids zero-size array ‘ctrs’
 make[3]: *** [libgcov-util.o] Error 1

 host compiler is gcc 4.3.4

 Richard.


 On my side, commit 212448 breaks the cross-build of GCC for targets
 using newlib:
 * arm-none-eabi
 * aarch64-none-elf
 /usr/include/sys/stat.h: In function 8098void
 gcov_output_files(const char*, gcov_info*)8099:
 /usr/include/sys/stat.h:317: error: too few arguments to function
 8098int mkdir(const char*, __mode_t)8099
 /tmp/1392958_1.tmpdir/aci-gcc-fsf/sources/gcc-fsf/gccsrc/gcc/gcov-tool.c:96:
 error: at this point in file
 make[2]: *** [gcov-tool.o] Error 1

 Christophe.

 Thanks,

 -Rong

 On Thu, May 1, 2014 at 3:37 PM, Rong Xu x...@google.com wrote:
 Hi, Honza,

 Please find the new patch in the attachment. This patch integrated
 your earlier suggestions. The noticeable changes are:
 (1) build specialized object for libgcov-driver.c and libgcov-merge.c
 and link into gcov-tool, rather including the source file.
 (2) split some gcov counter definition code to gcov-counter.def to
 avoid code duplication.
 (3) use a macro for gcov_read_counter(), so gcov-tool can use existing
 code in libgcov-merge.c with minimal change.

 This patch does not address the suggestion of creating a new directory
 for libgcov. I agree with you that's a much better
 and cleaner structure we should go for. We can do that in follow-up 
 patches.

 I tested this patch with boostrap and profiledbootstrap. Other tests
 are on-going.

 Thanks,

 -Rong

 On Wed, Apr 16, 2014 at 8:34 PM, Jan Hubicka hubi...@ucw.cz wrote:
 GCOT_TOOL needs to use this function to read the string in gcda file
 to memory to construct gcov_info objects.
 As you noticed, gcov runtime does not need this interface. But
 gcov-tool links with gcov runtime and it also uses the function.
 We could make it available in gcov_runtime, but that will slightly
 increase the memory footprint.

 Yep, it is not really pretty. I wrote bellow some plan how things may be
 structured in more convenient way.  Any work in that direction would be 
 welcome.

 The planned new functions for trunk version is: (1) overlap score b/w
 two profiles (2) better dumping (top hot objects/function/counters)
 and statistics.
 Once this basic version is in, we can start to add the new functionality.

 Sounds good. I assume the autoFDO does not go via gcov tool but rather 
 uses
 custom reader of profile data at GCC side?
 I wonder, are there any recent overview papers/slides/text of design of 
 AutoFDO
 and other features to be merged?
 I remember the talk from GCC Summit and I did read some of pre-LTO LIPO
 sources  papers, but it would be nice to have somethin up to date.

 libgcov-util.o is built in gcc/ directory, rather in libgcc.
 It's directly linked to gcov-tool.
 So libgcov-util.o is built for HOST, not TARGET.
 With makefile changes, we can built HOST version of libgcov-driver.o
 and libgcov-merge.o.
 I also need to make some static functions/variables public.

 I suppose that can go with IN_GCOV_TOOL ifdef.

 So we currently have basic gcov io handling in gcc/gcov-io.c that is 
 borrowed
 by libgcc/libgcov* code.  We also will get libgcov-util.c in libgcc 
 directory
 that is actually borrowed by by gcc/gcov-tool.c code.

 We now have one runtime using STDIO for streaming and kernel has custom 
 version
 that goes via /proc interface (last time I looked).  We added some 
 abstraction
 into libgcov-interface that is the part that relies on STDIO, partly via 
 gcov-io.c
 code and now we have in-memory handling code in libgcov-util.

 I guess it would make most sentse to put all the gcov code into a new 
 directory
 (libgcov) and make it stand-alone library that can be configured
 1) for stdio based runtime as we do not
 2) for runtime missing the interface and relyin on user providing it
 3) for use within gcov file manipulation tools with reorg of
 GCC/gcov/gcov-dump/gcov-tool to all use the same low-level interfaces.
 In a longer term, I would like to make FDO profiling intstrumentation to 
 happen
 at linktime. For that I need to make the instrumentation code (minimal 
 spaning
 tree  friends) to work w/o cgraph that would ideally be done in a shared
 implementation
  Won't this get wrong answer when counters[0] is not the same?
  I would expected the merging code to compare the counters first. 
  

Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Jakub Jelinek
On Fri, Jul 11, 2014 at 08:42:27AM -0700, Xinliang David Li wrote:
 I wonder why. The struct definition for gcov_fn_info has not changed
 in this patch.

Perhaps it has been used only in C until now?

Jakub


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Xinliang David Li
right.

Rong, the fix would be just change ctr array size to 1. For each
function, there should be at least one kind of counters -- see the
assertion in build_fn_info_type.  There are some code that do
'sizeof(gcov_fn_info)' when computing heap size -- they can be
adjusted or leave it as it is (if not doing memcpy for the whole
array).

David

On Fri, Jul 11, 2014 at 8:44 AM, Jakub Jelinek ja...@redhat.com wrote:
 On Fri, Jul 11, 2014 at 08:42:27AM -0700, Xinliang David Li wrote:
 I wonder why. The struct definition for gcov_fn_info has not changed
 in this patch.

 Perhaps it has been used only in C until now?

 Jakub


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Rong Xu
I should use _WIN32 macro. This code is for windows mkdir api.
I'll commit a patch for this shortly (approved by honza).

-Rong

On Fri, Jul 11, 2014 at 8:49 AM, Xinliang David Li davi...@google.com wrote:
 What is the macro to test POSIX IO on host? The guard uses
 TARGET_POSIX_IO which is not right.

 David

 On Fri, Jul 11, 2014 at 1:12 AM, Christophe Lyon
 christophe.l...@linaro.org wrote:
 On 11 July 2014 10:07, Richard Biener richard.guent...@gmail.com wrote:
 On Mon, May 5, 2014 at 7:17 PM, Rong Xu x...@google.com wrote:
 Here is the updated patch. The difference with patch set 3 is
 (1) use the gcov-counter.def for scaling operation.
 (2) fix wrong scaling function for time-profiler.

 passed with bootstrap, profiledboostrap and SPEC2006.

 One of the patches breaks bootstrap for me:

 /space/rguenther/src/svn/trunk/gcc/../libgcc/libgcov.h:184: error: ISO
 C++ forbids zero-size array ‘ctrs’
 make[3]: *** [libgcov-util.o] Error 1

 host compiler is gcc 4.3.4

 Richard.


 On my side, commit 212448 breaks the cross-build of GCC for targets
 using newlib:
 * arm-none-eabi
 * aarch64-none-elf
 /usr/include/sys/stat.h: In function 8098void
 gcov_output_files(const char*, gcov_info*)8099:
 /usr/include/sys/stat.h:317: error: too few arguments to function
 8098int mkdir(const char*, __mode_t)8099
 /tmp/1392958_1.tmpdir/aci-gcc-fsf/sources/gcc-fsf/gccsrc/gcc/gcov-tool.c:96:
 error: at this point in file
 make[2]: *** [gcov-tool.o] Error 1

 Christophe.

 Thanks,

 -Rong

 On Thu, May 1, 2014 at 3:37 PM, Rong Xu x...@google.com wrote:
 Hi, Honza,

 Please find the new patch in the attachment. This patch integrated
 your earlier suggestions. The noticeable changes are:
 (1) build specialized object for libgcov-driver.c and libgcov-merge.c
 and link into gcov-tool, rather including the source file.
 (2) split some gcov counter definition code to gcov-counter.def to
 avoid code duplication.
 (3) use a macro for gcov_read_counter(), so gcov-tool can use existing
 code in libgcov-merge.c with minimal change.

 This patch does not address the suggestion of creating a new directory
 for libgcov. I agree with you that's a much better
 and cleaner structure we should go for. We can do that in follow-up 
 patches.

 I tested this patch with boostrap and profiledbootstrap. Other tests
 are on-going.

 Thanks,

 -Rong

 On Wed, Apr 16, 2014 at 8:34 PM, Jan Hubicka hubi...@ucw.cz wrote:
 GCOT_TOOL needs to use this function to read the string in gcda file
 to memory to construct gcov_info objects.
 As you noticed, gcov runtime does not need this interface. But
 gcov-tool links with gcov runtime and it also uses the function.
 We could make it available in gcov_runtime, but that will slightly
 increase the memory footprint.

 Yep, it is not really pretty. I wrote bellow some plan how things may be
 structured in more convenient way.  Any work in that direction would be 
 welcome.

 The planned new functions for trunk version is: (1) overlap score b/w
 two profiles (2) better dumping (top hot objects/function/counters)
 and statistics.
 Once this basic version is in, we can start to add the new 
 functionality.

 Sounds good. I assume the autoFDO does not go via gcov tool but rather 
 uses
 custom reader of profile data at GCC side?
 I wonder, are there any recent overview papers/slides/text of design of 
 AutoFDO
 and other features to be merged?
 I remember the talk from GCC Summit and I did read some of pre-LTO LIPO
 sources  papers, but it would be nice to have somethin up to date.

 libgcov-util.o is built in gcc/ directory, rather in libgcc.
 It's directly linked to gcov-tool.
 So libgcov-util.o is built for HOST, not TARGET.
 With makefile changes, we can built HOST version of libgcov-driver.o
 and libgcov-merge.o.
 I also need to make some static functions/variables public.

 I suppose that can go with IN_GCOV_TOOL ifdef.

 So we currently have basic gcov io handling in gcc/gcov-io.c that is 
 borrowed
 by libgcc/libgcov* code.  We also will get libgcov-util.c in libgcc 
 directory
 that is actually borrowed by by gcc/gcov-tool.c code.

 We now have one runtime using STDIO for streaming and kernel has custom 
 version
 that goes via /proc interface (last time I looked).  We added some 
 abstraction
 into libgcov-interface that is the part that relies on STDIO, partly via 
 gcov-io.c
 code and now we have in-memory handling code in libgcov-util.

 I guess it would make most sentse to put all the gcov code into a new 
 directory
 (libgcov) and make it stand-alone library that can be configured
 1) for stdio based runtime as we do not
 2) for runtime missing the interface and relyin on user providing it
 3) for use within gcov file manipulation tools with reorg of
 GCC/gcov/gcov-dump/gcov-tool to all use the same low-level interfaces.
 In a longer term, I would like to make FDO profiling intstrumentation to 
 happen
 at linktime. For that I need to make the instrumentation code (minimal 
 spaning
 tree  friends) to 

Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Rong Xu
I did see the warning in the bootstrap, but it did not exit the build.
I thought it was ok.

I'll have a patch for this and send for review.

-Rong

On Fri, Jul 11, 2014 at 9:13 AM, Xinliang David Li davi...@google.com wrote:
 right.

 Rong, the fix would be just change ctr array size to 1. For each
 function, there should be at least one kind of counters -- see the
 assertion in build_fn_info_type.  There are some code that do
 'sizeof(gcov_fn_info)' when computing heap size -- they can be
 adjusted or leave it as it is (if not doing memcpy for the whole
 array).

 David

 On Fri, Jul 11, 2014 at 8:44 AM, Jakub Jelinek ja...@redhat.com wrote:
 On Fri, Jul 11, 2014 at 08:42:27AM -0700, Xinliang David Li wrote:
 I wonder why. The struct definition for gcov_fn_info has not changed
 in this patch.

 Perhaps it has been used only in C until now?

 Jakub


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Rong Xu
Richard,

I looked at my patch again. I already add -Wno-error to libgcov-util.o
compilation:
In line 200 of gcc/Makefile.in
libgcov-util.o-warn = -Wno-error

In my test, I used gcc-4.6 as the host compiler. I got warning like this:

In file included from ../../gcc/gcc/../libgcc/libgcov-util.c:30:0:
../../gcc/gcc/../libgcc/libgcov.h:184:30: warning: ISO C++ forbids
zero-size array ‘ctrs’ [-pedantic]

Can you check your buildlog to see if -Wno-error is added to the command line?

Thanks,

-Rong

On Fri, Jul 11, 2014 at 9:47 AM, Rong Xu x...@google.com wrote:
 I did see the warning in the bootstrap, but it did not exit the build.
 I thought it was ok.

 I'll have a patch for this and send for review.

 -Rong

 On Fri, Jul 11, 2014 at 9:13 AM, Xinliang David Li davi...@google.com wrote:
 right.

 Rong, the fix would be just change ctr array size to 1. For each
 function, there should be at least one kind of counters -- see the
 assertion in build_fn_info_type.  There are some code that do
 'sizeof(gcov_fn_info)' when computing heap size -- they can be
 adjusted or leave it as it is (if not doing memcpy for the whole
 array).

 David

 On Fri, Jul 11, 2014 at 8:44 AM, Jakub Jelinek ja...@redhat.com wrote:
 On Fri, Jul 11, 2014 at 08:42:27AM -0700, Xinliang David Li wrote:
 I wonder why. The struct definition for gcov_fn_info has not changed
 in this patch.

 Perhaps it has been used only in C until now?

 Jakub


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Jan Hubicka
 Richard,
 
 I looked at my patch again. I already add -Wno-error to libgcov-util.o
 compilation:
 In line 200 of gcc/Makefile.in
 libgcov-util.o-warn = -Wno-error
 
 In my test, I used gcc-4.6 as the host compiler. I got warning like this:
 
 In file included from ../../gcc/gcc/../libgcc/libgcov-util.c:30:0:
 ../../gcc/gcc/../libgcc/libgcov.h:184:30: warning: ISO C++ forbids
 zero-size array ???ctrs??? [-pedantic]
 
 Can you check your buildlog to see if -Wno-error is added to the command line?

I would preffer the libgcov.h (that is interface header to libgcov users) to be
valid C++, so we still ought to fix it.

Honza
 
 Thanks,
 
 -Rong
 
 On Fri, Jul 11, 2014 at 9:47 AM, Rong Xu x...@google.com wrote:
  I did see the warning in the bootstrap, but it did not exit the build.
  I thought it was ok.
 
  I'll have a patch for this and send for review.
 
  -Rong
 
  On Fri, Jul 11, 2014 at 9:13 AM, Xinliang David Li davi...@google.com 
  wrote:
  right.
 
  Rong, the fix would be just change ctr array size to 1. For each
  function, there should be at least one kind of counters -- see the
  assertion in build_fn_info_type.  There are some code that do
  'sizeof(gcov_fn_info)' when computing heap size -- they can be
  adjusted or leave it as it is (if not doing memcpy for the whole
  array).
 
  David
 
  On Fri, Jul 11, 2014 at 8:44 AM, Jakub Jelinek ja...@redhat.com wrote:
  On Fri, Jul 11, 2014 at 08:42:27AM -0700, Xinliang David Li wrote:
  I wonder why. The struct definition for gcov_fn_info has not changed
  in this patch.
 
  Perhaps it has been used only in C until now?
 
  Jakub


Re: [PATCH] offline gcda profile processing tool

2014-07-11 Thread Rong Xu
On Fri, Jul 11, 2014 at 11:46 AM, Jan Hubicka hubi...@ucw.cz wrote:
 Richard,

 I looked at my patch again. I already add -Wno-error to libgcov-util.o
 compilation:
 In line 200 of gcc/Makefile.in
 libgcov-util.o-warn = -Wno-error

 In my test, I used gcc-4.6 as the host compiler. I got warning like this:

 In file included from ../../gcc/gcc/../libgcc/libgcov-util.c:30:0:
 ../../gcc/gcc/../libgcc/libgcov.h:184:30: warning: ISO C++ forbids
 zero-size array ???ctrs??? [-pedantic]

 Can you check your buildlog to see if -Wno-error is added to the command 
 line?

 I would preffer the libgcov.h (that is interface header to libgcov users) to 
 be
 valid C++, so we still ought to fix it.

 Honza

OK. I will send out a patch for review.

-Rong


 Thanks,

 -Rong

 On Fri, Jul 11, 2014 at 9:47 AM, Rong Xu x...@google.com wrote:
  I did see the warning in the bootstrap, but it did not exit the build.
  I thought it was ok.
 
  I'll have a patch for this and send for review.
 
  -Rong
 
  On Fri, Jul 11, 2014 at 9:13 AM, Xinliang David Li davi...@google.com 
  wrote:
  right.
 
  Rong, the fix would be just change ctr array size to 1. For each
  function, there should be at least one kind of counters -- see the
  assertion in build_fn_info_type.  There are some code that do
  'sizeof(gcov_fn_info)' when computing heap size -- they can be
  adjusted or leave it as it is (if not doing memcpy for the whole
  array).
 
  David
 
  On Fri, Jul 11, 2014 at 8:44 AM, Jakub Jelinek ja...@redhat.com wrote:
  On Fri, Jul 11, 2014 at 08:42:27AM -0700, Xinliang David Li wrote:
  I wonder why. The struct definition for gcov_fn_info has not changed
  in this patch.
 
  Perhaps it has been used only in C until now?
 
  Jakub


Re: [PATCH] offline gcda profile processing tool

2014-05-25 Thread Jan Hubicka
   /* Size of the longest file name. */
  -static size_t gcov_max_filename = 0;
  +/* We need to expose this static variable when compiling for gcov-tool.  
  */
  +#ifndef IN_GCOV_TOOL
  +static
  +#endif
  +size_t gcov_max_filename = 0;
 
 
  Why max_filename needs to be exported?
 
 For code efficiency, we allocate the gi_filename buffer only once in
 the dumping, using the maximum filename length, which
 is set in gcov_init(). For gcov-tool, we don't have gcov_init, and we
 set the value when reading the gcda files.
 Now libgcov-driver.o is a separated compilation unit. I need to make
 then static gi_filename global to allow the access.

Note sure if code efficiency wins here over convolutin APIs, but I guess
it is OK.
Please update comment to explain how it is used, so we keep track of it
more easily.

 2014-05-20  Rong Xu  x...@google.com
 
   * gcc/gcov-io.c (gcov_position): Make avaialble to gcov-tool.
   (gcov_is_error): Ditto.
   (gcov_read_string): Ditto.
   (gcov_read_sync): Ditto.
   * gcc/gcov-io.h: Move counter defines to gcov-counter.def.
   * gcc/gcov-dump.c (tag_counters): Use gcov-counter.def.
   * gcc/coverage.c: Ditto.
   * gcc/gcov-tool.c: Offline gcda profile processing tool.
 (unlink_gcda_file): Remove one gcda file.
   (unlink_profile_dir): Remove gcda files from the profile path.
   (profile_merge): Merge two profiles in directory.
   (print_merge_usage_message): Print merge usage.
   (merge_usage): Print merge usage and exit.
   (do_merge): Driver for profile merge sub-command.
   (profile_rewrite): Rewrite profile.
   (print_rewrite_usage_message): Print rewrite usage.
   (rewrite_usage): Print rewrite usage and exit.
   (do_rewrite): Driver for profile rewrite sub-command.
   (print_usage): Print gcov-info usage and exit.
   (print_version): Print gcov-info version.
   (process_args): Process arguments.
   (main): Main routine for gcov-tool.
   * gcc/Makefile.in: Build and install gcov-tool.
   * gcc/gcov-counter.def: New file split from gcov-io.h.
   * libgcc/libgcov-driver.c (gcov_max_filename): Make available
 to gcov-tool.
   * libgcc/libgcov-merge.c (__gcov_merge_add): Replace
 gcov_read_counter() with a Macro.
   (__gcov_merge_ior): Ditto.
   (__gcov_merge_time_profile): Ditto.
   (__gcov_merge_single): Ditto.
   (__gcov_merge_delta): Ditto.
   * libgcc/libgcov-util.c (void gcov_set_verbose): Set the verbose flag
 in the utility functions.
   (set_fn_ctrs): Utility function for reading gcda files to in-memory
 gcov_list object link lists.
   (tag_function): Ditto.
   (tag_blocks): Ditto.
   (tag_arcs): Ditto.
   (tag_lines): Ditto.
   (tag_counters): Ditto.
   (tag_summary): Ditto.
   (read_gcda_finalize): Ditto.
   (read_gcda_file): Ditto.
   (ftw_read_file): Ditto.
   (read_profile_dir_init): Ditto.
   (gcov_read_profile_dir): Ditto.
   (gcov_read_counter_mem): Ditto.
   (gcov_get_merge_weight): Ditto.
   (merge_wrapper): A wrapper function that calls merging handler.
   (gcov_merge): Merge two gcov_info objects with weights.
   (find_match_gcov_info): Find the matched gcov_info in the list.
   (gcov_profile_merge): Merge two gcov_info object lists.
   (__gcov_add_counter_op): Process edge profile counter values.
   (__gcov_ior_counter_op): Process IOR profile counter values.
   (__gcov_delta_counter_op): Process delta profile counter values.
   (__gcov_single_counter_op): Process single  profile counter values.
   (fp_scale): Callback function for float-point scaling.
   (int_scale): Callback function for integer fraction scaling. 
   (gcov_profile_scale): Scaling profile counters.
   (gcov_profile_normalize): Normalize profile counters.
   * libgcc/libgcov.h: Add headers and functions for gcov-tool use.
 (gcov_get_counter): New.
 (gcov_get_counter_target): Ditto.
 (struct gcov_info): Make the functions field mutable in gcov-tool
 compilation.
   * gcc/doc/gcc.texi: Include gcov-tool.texi.
   * gcc/doc/gcov-tool.texi: Document for gcov-tool.

OK, with changes bellow.
I apologize for the delay - it is a long patch and I am bit swamped in tasks
these days.

 Index: gcc/gcov-tool.c
 ===
 --- gcc/gcov-tool.c   (revision 0)
 +++ gcc/gcov-tool.c   (revision 0)
 @@ -0,0 +1,466 @@
 +/* Gcc offline profile processing tool support. */
 +/* Compile this one with gcc.  */

What compile this one with gcc means? :)
 +
 +static int verbose;

Perhaps in C++ times, it could be bool and have comment in front of it
(per coding standards)
 +
 +/* Remove file NAME if it has a gcda suffix. */
 +
 +static int
 +unlink_gcda_file (const char *name,
 +  const struct stat *status ATTRIBUTE_UNUSED,
 +   

Re: [PATCH] offline gcda profile processing tool

2014-05-15 Thread Jan Hubicka
 Hi, Honza,
 
 Please find the new patch in the attachment. This patch integrated
 your earlier suggestions. The noticeable changes are:
 (1) build specialized object for libgcov-driver.c and libgcov-merge.c
 and link into gcov-tool, rather including the source file.
 (2) split some gcov counter definition code to gcov-counter.def to
 avoid code duplication.
 (3) use a macro for gcov_read_counter(), so gcov-tool can use existing
 code in libgcov-merge.c with minimal change.
 
 This patch does not address the suggestion of creating a new directory
 for libgcov. I agree with you that's a much better
 and cleaner structure we should go for. We can do that in follow-up patches.

Yep, lets do this incrementally. thanks!
 
 I tested this patch with boostrap and profiledbootstrap. Other tests
 are on-going.
 
 Thanks,
 
 -Rong
 2014-05-01  Rong Xu  x...@google.com
 
   * gcc/gcov-io.c (gcov_position): Make avaialble to gcov-tool.
   (gcov_is_error): Ditto.
   (gcov_read_string): Ditto.
   (gcov_read_sync): Ditto.
   * gcc/gcov-io.h: Move counter defines to gcov-counter.def.
   * gcc/gcov-dump.c (tag_counters): Use gcov-counter.def.
   * gcc/coverage.c: Ditto.
   * gcc/gcov-tool.c: Offline gcda profile processing tool.
 (unlink_gcda_file): Remove one gcda file.
   (unlink_profile_dir): Remove gcda files from the profile path.
   (profile_merge): Merge two profiles in directory.
   (print_merge_usage_message): Print merge usage.
   (merge_usage): Print merge usage and exit.
   (do_merge): Driver for profile merge sub-command.
   (profile_rewrite): Rewrite profile.
   (print_rewrite_usage_message): Print rewrite usage.
   (rewrite_usage): Print rewrite usage and exit.
   (do_rewrite): Driver for profile rewrite sub-command.
   (print_usage): Print gcov-info usage and exit.
   (print_version): Print gcov-info version.
   (process_args): Process arguments.
   (main): Main routine for gcov-tool.
   * gcc/Makefile.in: Build and install gcov-tool.
   * gcc/gcov-counter.def: New file split from gcov-io.h.
   * libgcc/libgcov-driver.c (gcov_max_filename): Make available
 to gcov-tool.
   * libgcc/libgcov-merge.c (__gcov_merge_add): Replace
 gcov_read_counter() with a Macro.
   (__gcov_merge_ior): Ditto.
   (__gcov_merge_time_profile): Ditto.
   (__gcov_merge_single): Ditto.
   (__gcov_merge_delta): Ditto.
   * libgcc/libgcov-util.c (void gcov_set_verbose): Set the verbose flag
 in the utility functions.
   (set_fn_ctrs): Utility function for reading gcda files to in-memory
 gcov_list object link lists.
   (tag_function): Ditto.
   (tag_blocks): Ditto.
   (tag_arcs): Ditto.
   (tag_lines): Ditto.
   (tag_counters): Ditto.
   (tag_summary): Ditto.
   (read_gcda_finalize): Ditto.
   (read_gcda_file): Ditto.
   (ftw_read_file): Ditto.
   (read_profile_dir_init): Ditto.
   (gcov_read_profile_dir): Ditto.
   (gcov_read_counter_mem): Ditto.
   (gcov_get_merge_weight): Ditto.
   (merge_wrapper): A wrapper function that calls merging handler.
   (gcov_merge): Merge two gcov_info objects with weights.
   (find_match_gcov_info): Find the matched gcov_info in the list.
   (gcov_profile_merge): Merge two gcov_info object lists.
   (__gcov_add_counter_op): Process edge profile counter values.
   (__gcov_ior_counter_op): Process IOR profile counter values.
   (__gcov_delta_counter_op): Process delta profile counter values.
   (__gcov_single_counter_op): Process single  profile counter values.
   (fp_scale): Callback function for float-point scaling.
   (int_scale): Callback function for integer fraction scaling. 
   (gcov_profile_scale): Scaling profile counters.
   (gcov_profile_normalize): Normalize profile counters.
   * libgcc/libgcov.h: Add headers and macros for gcov-tool use.
 (GCOV_GET_COUNTER): New.
 (GCOV_GET_COUNTER_TARGET): Ditto.
 (struct gcov_info): Make the functions field mutable in gcov-tool
 compilation.
 
 Index: gcc/gcov-io.c
 ===
 --- gcc/gcov-io.c (revision 209981)
 +++ gcc/gcov-io.c (working copy)
 @@ -64,7 +64,11 @@ GCOV_LINKAGE struct gcov_var
  } gcov_var;
  
  /* Save the current position in the gcov file.  */
 -static inline gcov_position_t
 +/* We need to expose this function when compiling for gcov-tool.  */
 +#ifndef IN_GCOV_TOOL
 +static inline
 +#endif
 +gcov_position_t
  gcov_position (void)
  {
gcc_assert (gcov_var.mode  0); 
 @@ -72,7 +76,11 @@ gcov_position (void)
  }
  
  /* Return nonzero if the error flag is set.  */
 -static inline int 
 +/* We need to expose this function when compiling for gcov-tool.  */
 +#ifndef IN_GCOV_TOOL
 +static inline
 +#endif
 +int

I am still not too happy about the ifdef noise here, but I suppose 

Re: [PATCH] offline gcda profile processing tool

2014-04-16 Thread Rong Xu
On Tue, Apr 15, 2014 at 2:38 PM, Jan Hubicka hubi...@ucw.cz wrote:
 Rong, David, Dehao, Teresa
 I would like to have some rought idea of what we could merge this stage1. 
 There is
 certainly a lot of interesting stuff on the google branch including AutoFDO, 
 LIPO,
 the multivalue profile counters that may be used by the new devirtualization 
 bits
 and more. I also think we should switch counts into floating point 
 representation
 so Teresa's splitting patch works.

 Can we get plans to make this effective? My personal schedule is quite free 
 until
 April 29 when I go to Czech Republic for wedding and I will be back in Calgary
 at 14th.

 2014-03-03  Rong Xu  x...@google.com

   * gcc/gcov-io.c (gcov_read_string): Make this routine available
 to gcov-tool.
   (gcov_sync): Ditto.
   * gcc/Makefile.in: Build and install gcov-tool.
   * gcc/gcov-tool.c (unlink_gcda_file): Remove one gcda file.
   (unlink_profile_dir): Remove gcda files from the profile path.
   (profile_merge): Merge two profiles in directory.
   (print_merge_usage_message): Print merge usage.
   (merge_usage): Print merge usage and exit.
   (do_merge): Driver for profile merge sub-command.
   (profile_rewrite): Rewrite profile.
   (print_rewrite_usage_message): Print rewrite usage.
   (rewrite_usage): Print rewrite usage and exit.
   (do_rewrite): Driver for profile rewrite sub-command.
   (print_usage): Print gcov-info usage and exit.
   (print_version): Print gcov-info version.
   (process_args): Process arguments.
   (main): Main routine for gcov-tool.
   * libgcc/libgcov.h : Include the set of base-type headers for
 gcov-tool.
 (struct gcov_info): Make the functions field mutable in gcov-tool
 compilation.
   * libgcc/libgcov-merge.c (gcov_get_counter): New wrapper function
 to get the profile counter.
   (gcov_get_counter_target): New wrapper function to get the profile
 values that should not be scaled.
   (__gcov_merge_add): Replace gcov_read_counter() with the wrapper
 functions.
   (__gcov_merge_ior): Ditto.
   (__gcov_merge_time_profile): Ditto.
   (__gcov_merge_single): Ditto.
   (__gcov_merge_delta): Ditto.
   * libgcc/libgcov-util.c (void gcov_set_verbose): Set the verbose flag
 in the utility functions.
   (set_fn_ctrs): Utility function for reading gcda files to in-memory
 gcov_list object link lists.
   (tag_function): Ditto.
   (tag_blocks): Ditto.
   (tag_arcs): Ditto.
   (tag_lines): Ditto.
   (tag_counters): Ditto.
   (tag_summary): Ditto.
   (read_gcda_finalize): Ditto.
   (read_gcda_file): Ditto.
   (ftw_read_file): Ditto.
   (read_profile_dir_init): Ditto.
   (gcov_read_profile_dir): Ditto.
   (gcov_read_counter_mem): Ditto.
   (gcov_get_merge_weight): Ditto.
   (merge_wrapper): A wrapper function that calls merging handler.
   (gcov_merge): Merge two gcov_info objects with weights.
   (find_match_gcov_info): Find the matched gcov_info in the list.
   (gcov_profile_merge): Merge two gcov_info object lists.
   (__gcov_add_counter_op): Process edge profile counter values.
   (__gcov_ior_counter_op): Process IOR profile counter values.
   (__gcov_delta_counter_op): Process delta profile counter values.
   (__gcov_single_counter_op): Process single  profile counter values.
   (fp_scale): Callback function for float-point scaling.
   (int_scale): Callback function for integer fraction scaling.
   (gcov_profile_scale): Scaling profile counters.
   (gcov_profile_normalize): Normalize profile counters.

 Index: gcc/gcov-io.c
 ===
 --- gcc/gcov-io.c (revision 208237)
 +++ gcc/gcov-io.c (working copy)
 @@ -564,7 +564,7 @@ gcov_read_counter (void)
 buffer, or NULL on empty string. You must copy the string before
 calling another gcov function.  */

 -#if !IN_LIBGCOV
 +#if !IN_LIBGCOV || defined (IN_GCOV_TOOL)
  GCOV_LINKAGE const char *
  gcov_read_string (void)
  {
 @@ -641,7 +641,7 @@ gcov_read_summary (struct gcov_summary *summary)
  }
  }

 -#if !IN_LIBGCOV
 +#if !IN_LIBGCOV || defined (IN_GCOV_TOOL)
  /* Reset to a known position.  BASE should have been obtained from
 gcov_position, LENGTH should be a record length.  */

 I am slightly confused here, IN_LIBGCOV IMO means that the gcov-io is going
 to be linked into the gcov runtime as opposed to gcc, gcov, gcov-dump or
 gcov-tool.  Why we define IN_LIBGCOV  IN_GCOV_TOOL?

GCOT_TOOL needs to use this function to read the string in gcda file
to memory to construct gcov_info objects.
As you noticed, gcov runtime does not need this interface. But
gcov-tool links with gcov runtime and it also uses the function.
We could make it available in gcov_runtime, but that will slightly
increase the memory footprint.

 

Re: [PATCH] offline gcda profile processing tool

2014-04-16 Thread Jan Hubicka
 GCOT_TOOL needs to use this function to read the string in gcda file
 to memory to construct gcov_info objects.
 As you noticed, gcov runtime does not need this interface. But
 gcov-tool links with gcov runtime and it also uses the function.
 We could make it available in gcov_runtime, but that will slightly
 increase the memory footprint.

Yep, it is not really pretty. I wrote bellow some plan how things may be
structured in more convenient way.  Any work in that direction would be welcome.
 
 The planned new functions for trunk version is: (1) overlap score b/w
 two profiles (2) better dumping (top hot objects/function/counters)
 and statistics.
 Once this basic version is in, we can start to add the new functionality.

Sounds good. I assume the autoFDO does not go via gcov tool but rather uses
custom reader of profile data at GCC side?
I wonder, are there any recent overview papers/slides/text of design of AutoFDO
and other features to be merged?
I remember the talk from GCC Summit and I did read some of pre-LTO LIPO
sources  papers, but it would be nice to have somethin up to date.
 
 libgcov-util.o is built in gcc/ directory, rather in libgcc.
 It's directly linked to gcov-tool.
 So libgcov-util.o is built for HOST, not TARGET.
 With makefile changes, we can built HOST version of libgcov-driver.o
 and libgcov-merge.o.
 I also need to make some static functions/variables public.

I suppose that can go with IN_GCOV_TOOL ifdef.

So we currently have basic gcov io handling in gcc/gcov-io.c that is borrowed
by libgcc/libgcov* code.  We also will get libgcov-util.c in libgcc directory
that is actually borrowed by by gcc/gcov-tool.c code.

We now have one runtime using STDIO for streaming and kernel has custom version
that goes via /proc interface (last time I looked).  We added some abstraction
into libgcov-interface that is the part that relies on STDIO, partly via 
gcov-io.c
code and now we have in-memory handling code in libgcov-util.

I guess it would make most sentse to put all the gcov code into a new directory
(libgcov) and make it stand-alone library that can be configured 
1) for stdio based runtime as we do not
2) for runtime missing the interface and relyin on user providing it
3) for use within gcov file manipulation tools with reorg of
GCC/gcov/gcov-dump/gcov-tool to all use the same low-level interfaces.
In a longer term, I would like to make FDO profiling intstrumentation to happen
at linktime. For that I need to make the instrumentation code (minimal spaning
tree  friends) to work w/o cgraph that would ideally be done in a shared
implementation
  Won't this get wrong answer when counters[0] is not the same?
  I would expected the merging code to compare the counters first. Similarly 
  for delta counter.
 
 These *_op functions are for scaling only. So there is only one
 profile involved (thus there is no comparison).
 The merge handles are in libgcov-merge.c which have the code to handle
 mismatched profile targets.

I see, OK then.
 
  Adding correct rounding may actually make difference for Martin's startup 
  time work.
 
 Do you mean to use something like in RDIV macro?

Yes.

Honza


Re: [PATCH] offline gcda profile processing tool

2014-04-15 Thread Jan Hubicka
Rong, David, Dehao, Teresa
I would like to have some rought idea of what we could merge this stage1. There 
is
certainly a lot of interesting stuff on the google branch including AutoFDO, 
LIPO,
the multivalue profile counters that may be used by the new devirtualization 
bits
and more. I also think we should switch counts into floating point 
representation
so Teresa's splitting patch works.

Can we get plans to make this effective? My personal schedule is quite free 
until
April 29 when I go to Czech Republic for wedding and I will be back in Calgary
at 14th.

 2014-03-03  Rong Xu  x...@google.com
 
   * gcc/gcov-io.c (gcov_read_string): Make this routine available
 to gcov-tool.
   (gcov_sync): Ditto.
   * gcc/Makefile.in: Build and install gcov-tool. 
   * gcc/gcov-tool.c (unlink_gcda_file): Remove one gcda file.
   (unlink_profile_dir): Remove gcda files from the profile path.
   (profile_merge): Merge two profiles in directory.
   (print_merge_usage_message): Print merge usage.
   (merge_usage): Print merge usage and exit.
   (do_merge): Driver for profile merge sub-command.
   (profile_rewrite): Rewrite profile.
   (print_rewrite_usage_message): Print rewrite usage.
   (rewrite_usage): Print rewrite usage and exit.
   (do_rewrite): Driver for profile rewrite sub-command.
   (print_usage): Print gcov-info usage and exit.
   (print_version): Print gcov-info version.
   (process_args): Process arguments.
   (main): Main routine for gcov-tool.
   * libgcc/libgcov.h : Include the set of base-type headers for
 gcov-tool.
 (struct gcov_info): Make the functions field mutable in gcov-tool
 compilation.
   * libgcc/libgcov-merge.c (gcov_get_counter): New wrapper function
 to get the profile counter.
   (gcov_get_counter_target): New wrapper function to get the profile
 values that should not be scaled.
   (__gcov_merge_add): Replace gcov_read_counter() with the wrapper
 functions.
   (__gcov_merge_ior): Ditto.
   (__gcov_merge_time_profile): Ditto.
   (__gcov_merge_single): Ditto.
   (__gcov_merge_delta): Ditto.
   * libgcc/libgcov-util.c (void gcov_set_verbose): Set the verbose flag
 in the utility functions.
   (set_fn_ctrs): Utility function for reading gcda files to in-memory
 gcov_list object link lists.
   (tag_function): Ditto.
   (tag_blocks): Ditto.
   (tag_arcs): Ditto.
   (tag_lines): Ditto.
   (tag_counters): Ditto.
   (tag_summary): Ditto.
   (read_gcda_finalize): Ditto.
   (read_gcda_file): Ditto.
   (ftw_read_file): Ditto.
   (read_profile_dir_init): Ditto.
   (gcov_read_profile_dir): Ditto.
   (gcov_read_counter_mem): Ditto.
   (gcov_get_merge_weight): Ditto.
   (merge_wrapper): A wrapper function that calls merging handler.
   (gcov_merge): Merge two gcov_info objects with weights.
   (find_match_gcov_info): Find the matched gcov_info in the list.
   (gcov_profile_merge): Merge two gcov_info object lists.
   (__gcov_add_counter_op): Process edge profile counter values.
   (__gcov_ior_counter_op): Process IOR profile counter values.
   (__gcov_delta_counter_op): Process delta profile counter values.
   (__gcov_single_counter_op): Process single  profile counter values.
   (fp_scale): Callback function for float-point scaling.
   (int_scale): Callback function for integer fraction scaling. 
   (gcov_profile_scale): Scaling profile counters.
   (gcov_profile_normalize): Normalize profile counters.
 
 Index: gcc/gcov-io.c
 ===
 --- gcc/gcov-io.c (revision 208237)
 +++ gcc/gcov-io.c (working copy)
 @@ -564,7 +564,7 @@ gcov_read_counter (void)
 buffer, or NULL on empty string. You must copy the string before
 calling another gcov function.  */
  
 -#if !IN_LIBGCOV
 +#if !IN_LIBGCOV || defined (IN_GCOV_TOOL)
  GCOV_LINKAGE const char *
  gcov_read_string (void)
  {
 @@ -641,7 +641,7 @@ gcov_read_summary (struct gcov_summary *summary)
  }
  }
  
 -#if !IN_LIBGCOV
 +#if !IN_LIBGCOV || defined (IN_GCOV_TOOL)
  /* Reset to a known position.  BASE should have been obtained from
 gcov_position, LENGTH should be a record length.  */

I am slightly confused here, IN_LIBGCOV IMO means that the gcov-io is going
to be linked into the gcov runtime as opposed to gcc, gcov, gcov-dump or
gcov-tool.  Why we define IN_LIBGCOV  IN_GCOV_TOOL?
 Index: gcc/gcov-tool.c
 ===
 --- gcc/gcov-tool.c   (revision 0)
 +++ gcc/gcov-tool.c   (revision 0)
 @@ -0,0 +1,465 @@
 +/* Gcc offline profile processing tool support. */
 +/* Compile this one with gcc.  */
 +/* Copyright (C) 2014 Free Software Foundation, Inc.
 +   Contributed by Rong Xu x...@google.com.
 +
 +This file is part of GCC.
 +
 +GCC is free 

Re: [PATCH] offline gcda profile processing tool

2014-01-16 Thread Rong Xu
Ping.

-Rong

On Mon, Jan 13, 2014 at 12:43 PM, Rong Xu x...@google.com wrote:
 Hi,

 This patch implements gcov-tool, a offline profile processing tool.
 This version supports merging two profiles with weights, and scaling
 the profile with a floating-point / fraction weight.

 Earlier discussion can be found
 http://gcc.gnu.org/ml/gcc-patches/2013-11/msg02631.html

 In the discussion of the earlier patch, Honza wanted a separated
 patch for interface change for gcov_merge_*() functions. In this patch,
 the changes for gcov_merge_*() are minimal by using a wrapper function.
 So I include everything in this patch.

 Tested with bootstrap profiledboostrap and SPEC2006 profiles.

 Thanks,

 -Rong