Re: State of AutoFDO in GCC
On Mon, Apr 26, 2021 at 11:00 AM Andi Kleen wrote: > >There are multiple directional changes in this new tool: > >1) it uses perf-script trace output (in text) as input profile data; > > I suspect this will break regularly too > > (I personally did numerous changes to perf script output, and also > wrote a lot of parsing scripts) > > The perf script output has some bad problems, e.g. for file names or > processes > with spaces and some other issues. To make it handleable would need some > redesign to actually generate in a machine friendly format. > Andi, thanks for the input. +authors of the llvm-profgen tool for their experience with using perf script output. David > > A perf.data parser should be fine, just don't fill it up with asserts > and "be liberal what you accept" and ignore unknown records. > > -Andi >
Re: State of AutoFDO in GCC
On Mon, Apr 26, 2021 at 8:12 AM Andi Kleen wrote: > Jan Hubicka writes: > > > > Is there a way to get this working w/o using older perf? > > It's usually rather simple to fix up autofdo for new perf. > I did it before here > > https://github.com/andikleen/autofdo/commits/perf4-3 > > I think it would work always if it just ignored unknown records > (which is quite possible). perf adds new records now and then, > but they can be usually ignored by old tools. > Only the assert happy coding style in autofdo prevents it. > > BTW longer term my feeling is autofdo should be replaced with something > like https://lists.llvm.org/pipermail/llvm-dev/2020-August/144101.html > I think that would fix most of the weirdnesses in the current autofdo > implementation. > There are multiple directional changes in this new tool: 1) it uses perf-script trace output (in text) as input profile data; 2) it uses pseudo probe like instrumentation FDO to do profile matching; 3) it supports full profile context sensitivity (not just from inline stacks). #1 is most relevant to this discussion -- using text format reduces the pain introduced by perf binary format change. David > > -Andi >
Re: State of AutoFDO in GCC
On Sun, Apr 25, 2021 at 12:07 PM Jan Hubicka wrote: > David, > > > > The text format is documented here: > > https://clang.llvm.org/docs/UsersManual.html > > The binary format is not documented. The binary format is not guaranteed > to > > be backward compatible, so sharing the same format may not be the best > way > > as changes for clang may break GCC. > > > > Since linux perf format does not change, the tool should be relatively > > stable with low maintenance cost. Changes are needed only when some new > > AutoFDO features are added to the compiler side. > > I was under impression that it is indeed problem with the tool requiring > old format of linux perf. At least with opensuse distro the shipped tool > fails for me: > jan@skylake:~> create_llvm_prof --binary=./code --out=code.prof > E0425 21:01:55.038128 17977 perf_reader.cc:996] Unsupported event type > 79 > F0425 21:01:55.038295 17977 perf_parser.cc:240] Check failed: > reader_.ReadPerfSampleInfo(*parsed_event.raw_event, _info) > *** Check failure stack trace: *** > @ 0x55e6deb6058e (unknown) > @ 0x55e6deb94a49 (unknown) > .. > Aborted (core dumped) > > I collect data as intstructed here: > https://clang.llvm.org/docs/UsersManual.html > > It is from package autofdo-0.18-4.4.x86_64 and perf 5.11.15. > > Is there a way to get this working w/o using older perf? > Honza > > Interesting. That means we will also see the same error when using the latest perf. Wei, are you aware of the issue? David > > > Does LLVM's auto-FDO support non-Intel CPUs these days? > > > > > > > It supports LBR like events, so it is CPU vendor dependent. For ARM, > using > > ETM can achieve the goal, but I don't have detailed knowledge of it. > > > > David > > > > > > > > Honza > > > > > > > > David > > > > > > > > > > > > > > >> Honza > > > > >> > > > > > >> > David > > > > >> > > > > > >> > > > > > >> > > Honza > > > > >> > > > > > > > >> > > > David > > > > >> > > > > > > > >> > > > > > > > > >> > > > > Thoughts? > > > > >> > > > > Martin > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > Having the tool third-party makes keeping the whole > chain > > > > >> working > > > > >> > > more > > > > >> > > > > > difficult. > > > > >> > > > > > > > > > >> > > > > > Richard. > > > > >> > > > > > > > > > >> > > > > >> Thanks, > > > > >> > > > > >> > > > > >> > > > > >> David > > > > >> > > > > >> > > > > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka < > > > hubi...@ucw.cz> > > > > >> wrote: > > > > >> > > > > >> > > > > >> > > > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > > > > >> > > > > > GCC documentation for AutoFDO points to create_gcov > tool > > > > >> that > > > > >> > > > > converts > > > > >> > > > > >>> perf.data file into gcov format that can be consumed > by > > > gcc > > > > >> with > > > > >> > > > > >>> -fauto-profile ( > > > > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > > > > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > > > > >> > > > > > > > > > >> > > > > > I noticed that the source code for create_gcov has > been > > > > >> deleted > > > > >> > > from > > > > >> > > > > >>> https://github.com/google/autofdo on April 7. I asked > > > about > > > > >> that > > > > >> > > > > change > > > > >> > > > > >>> in that repo and got the following reply: > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738 > > > > >> > > > > > > > > > >> > > > > > "Actually we didn't use create_gcov and havn't > updated > > > > >> > > create_gcov > > > > >> > > > > for > > > > >> > > > > >>> years, and we also didn't have enough tests to > guarantee > > > it > > > > >> works > > > > >> > > (It > > > > >> > > > > was > > > > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you > > > need > > > > >> it, it > > > > >> > > is > > > > >> > > > > >>> welcomed to update create_gcov and add it to the > > > respository." > > > > >> > > > > > > > > > >> > > > > > Does this mean that AutoFDO is currently dead in > gcc? > > > > >> > > > > > > > > >> > > > > Hello. > > > > >> > > > > > > > > >> > > > > Yes. I know that even basic test cases have been > broken > > > for > > > > >> years > > > > >> > > in > > > > >> > > > > the > > > > >> > > > > >>> GCC. > > > > >> > > > > It's new to me that create_gcov was removed. > > > > >> > > > > > > > > >> > > > > I tend to send patch to GCC that will remove AutoFDO > from > > > > >> GCC. > > > > >> > > > > I known Bin spent some time working on AutoFDO, has > he > > > came > > > > >> up to > > > > >> > > > > >>> something? > > > > >> > > > > >>> > > > > >> > > > > >>> The GCC side of auto-FDO is not that hard. We have > most > > > of > > > > >> > > > > >>> infrastructure in place, but stopping point for me was > > > always > > > > >> > > > > difficulty > > > > >> > > > > >>> to get
Re: State of AutoFDO in GCC
On Fri, Apr 23, 2021 at 12:28 PM Jan Hubicka wrote: > > On Fri, Apr 23, 2021 at 10:27 AM Xinliang David Li > > wrote: > > > > > > > > > > > On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka wrote: > > > > > >> > > > >> > It uses create_llvm_prof tool which is in the same git repo. The > data > > >> > parsing part is shared with create_gcov, but the writer is obviously > > >> > different for the two tools. > > >> > > >> OK and what are the main differences between llvmand gcc format? > > >> > > >> GCC expects GCOV format, I think while LLVM uses a newly designed > binary > > > format. > > > > > > > > Sorry I missed your next reply. I forgot about the details of GCC' > format. > > Thanks for explanation. How hard would be to make GCC consume this > format? Is is reasonably stable and documented somewhere? > The text format is documented here: https://clang.llvm.org/docs/UsersManual.html The binary format is not documented. The binary format is not guaranteed to be backward compatible, so sharing the same format may not be the best way as changes for clang may break GCC. Since linux perf format does not change, the tool should be relatively stable with low maintenance cost. Changes are needed only when some new AutoFDO features are added to the compiler side. Does LLVM's auto-FDO support non-Intel CPUs these days? > It supports LBR like events, so it is CPU vendor dependent. For ARM, using ETM can achieve the goal, but I don't have detailed knowledge of it. David > > Honza > > > > David > > > > > > > > >> Honza > > >> > > > >> > David > > >> > > > >> > > > >> > > Honza > > >> > > > > > >> > > > David > > >> > > > > > >> > > > > > > >> > > > > Thoughts? > > >> > > > > Martin > > >> > > > > > > >> > > > > > > > >> > > > > > Having the tool third-party makes keeping the whole chain > > >> working > > >> > > more > > >> > > > > > difficult. > > >> > > > > > > > >> > > > > > Richard. > > >> > > > > > > > >> > > > > >> Thanks, > > >> > > > > >> > > >> > > > > >> David > > >> > > > > >> > > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka < > hubi...@ucw.cz> > > >> wrote: > > >> > > > > >> > > >> > > > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > > >> > > > > > GCC documentation for AutoFDO points to create_gcov tool > > >> that > > >> > > > > converts > > >> > > > > >>> perf.data file into gcov format that can be consumed by > gcc > > >> with > > >> > > > > >>> -fauto-profile ( > > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > > >> > > > > > > > >> > > > > > I noticed that the source code for create_gcov has been > > >> deleted > > >> > > from > > >> > > > > >>> https://github.com/google/autofdo on April 7. I asked > about > > >> that > > >> > > > > change > > >> > > > > >>> in that repo and got the following reply: > > >> > > > > > > > >> > > > > > > > >> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738 > > >> > > > > > > > >> > > > > > "Actually we didn't use create_gcov and havn't updated > > >> > > create_gcov > > >> > > > > for > > >> > > > > >>> years, and we also didn't have enough tests to guarantee > it > > >> works > > >> > > (It > > >> > > > > was > > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you > need > > >> it, it > > >> > > is > > >> > > > > >>> welcomed to update create_gcov and add it to the > respository." > > >> > > > > > > > >> > > > > > Does this mean that AutoFDO is currently dead in gcc? > > >> > > > > > > >> > > > > Hello. > > >> > > > > > > >> > > > > Yes. I know that even basic test cases have been broken > for > > >> years > > >> > > in > > >> > > > > the > > >> > > > > >>> GCC. > > >> > > > > It's new to me that create_gcov was removed. > > >> > > > > > > >> > > > > I tend to send patch to GCC that will remove AutoFDO from > > >> GCC. > > >> > > > > I known Bin spent some time working on AutoFDO, has he > came > > >> up to > > >> > > > > >>> something? > > >> > > > > >>> > > >> > > > > >>> The GCC side of auto-FDO is not that hard. We have most > of > > >> > > > > >>> infrastructure in place, but stopping point for me was > always > > >> > > > > difficulty > > >> > > > > >>> to get gcov-tool working. If some maintainer steps up, I > > >> think I > > >> > > can > > >> > > > > >>> fix GCC side. > > >> > > > > >>> > > >> > > > > >>> I am bit unsure how important feature it is - we have FDO > that > > >> > > works > > >> > > > > >>> quite well for most users but I know there are some users > of > > >> the > > >> > > LLVM > > >> > > > > >>> implementation and there is potential to tie this with > other > > >> > > hardware > > >> > > > > >>> events to asist i.e. if conversion (where one wants to > know > > >> how > > >> > > well > > >> > > > > CPU > > >> > > > > >>> predicts the jump rather than just the jump probability) > > >> which I > > >> > > always
Re: State of AutoFDO in GCC
On Fri, Apr 23, 2021 at 10:27 AM Xinliang David Li wrote: > > > On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka wrote: > >> > >> > It uses create_llvm_prof tool which is in the same git repo. The data >> > parsing part is shared with create_gcov, but the writer is obviously >> > different for the two tools. >> >> OK and what are the main differences between llvmand gcc format? >> >> GCC expects GCOV format, I think while LLVM uses a newly designed binary > format. > > Sorry I missed your next reply. I forgot about the details of GCC' format. David > > >> Honza >> > >> > David >> > >> > >> > > Honza >> > > > >> > > > David >> > > > >> > > > > >> > > > > Thoughts? >> > > > > Martin >> > > > > >> > > > > > >> > > > > > Having the tool third-party makes keeping the whole chain >> working >> > > more >> > > > > > difficult. >> > > > > > >> > > > > > Richard. >> > > > > > >> > > > > >> Thanks, >> > > > > >> >> > > > > >> David >> > > > > >> >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka >> wrote: >> > > > > >> >> > > > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: >> > > > > > GCC documentation for AutoFDO points to create_gcov tool >> that >> > > > > converts >> > > > > >>> perf.data file into gcov format that can be consumed by gcc >> with >> > > > > >>> -fauto-profile ( >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial). >> > > > > > >> > > > > > I noticed that the source code for create_gcov has been >> deleted >> > > from >> > > > > >>> https://github.com/google/autofdo on April 7. I asked about >> that >> > > > > change >> > > > > >>> in that repo and got the following reply: >> > > > > > >> > > > > > >> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738 >> > > > > > >> > > > > > "Actually we didn't use create_gcov and havn't updated >> > > create_gcov >> > > > > for >> > > > > >>> years, and we also didn't have enough tests to guarantee it >> works >> > > (It >> > > > > was >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you need >> it, it >> > > is >> > > > > >>> welcomed to update create_gcov and add it to the respository." >> > > > > > >> > > > > > Does this mean that AutoFDO is currently dead in gcc? >> > > > > >> > > > > Hello. >> > > > > >> > > > > Yes. I know that even basic test cases have been broken for >> years >> > > in >> > > > > the >> > > > > >>> GCC. >> > > > > It's new to me that create_gcov was removed. >> > > > > >> > > > > I tend to send patch to GCC that will remove AutoFDO from >> GCC. >> > > > > I known Bin spent some time working on AutoFDO, has he came >> up to >> > > > > >>> something? >> > > > > >>> >> > > > > >>> The GCC side of auto-FDO is not that hard. We have most of >> > > > > >>> infrastructure in place, but stopping point for me was always >> > > > > difficulty >> > > > > >>> to get gcov-tool working. If some maintainer steps up, I >> think I >> > > can >> > > > > >>> fix GCC side. >> > > > > >>> >> > > > > >>> I am bit unsure how important feature it is - we have FDO that >> > > works >> > > > > >>> quite well for most users but I know there are some users of >> the >> > > LLVM >> > > > > >>> implementation and there is potential to tie this with other >> > > hardware >> > > > > >>> events to asist i.e. if conversion (where one wants to know >> how >> > > well >> > > > > CPU >> > > > > >>> predicts the jump rather than just the jump probability) >> which I >> > > always >> > > > > >>> found potentially interesting. >> > > > > >>> >> > > > > >>> Honza >> > > > > >> > > > > Martin >> > > > > >> > > > > > >> > > > > > Thanks, >> > > > > > >> > > > > > Eugene >> > > > > > >> > > > > >> > > > > >>> >> > > > > >> > > > > >> > > >> >
Re: State of AutoFDO in GCC
On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka wrote: > > > > It uses create_llvm_prof tool which is in the same git repo. The data > > parsing part is shared with create_gcov, but the writer is obviously > > different for the two tools. > > OK and what are the main differences between llvmand gcc format? > > GCC expects GCOV format, I think while LLVM uses a newly designed binary format. David > Honza > > > > David > > > > > > > Honza > > > > > > > > David > > > > > > > > > > > > > > Thoughts? > > > > > Martin > > > > > > > > > > > > > > > > > Having the tool third-party makes keeping the whole chain working > > > more > > > > > > difficult. > > > > > > > > > > > > Richard. > > > > > > > > > > > >> Thanks, > > > > > >> > > > > > >> David > > > > > >> > > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka > wrote: > > > > > >> > > > > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > > > > > > GCC documentation for AutoFDO points to create_gcov tool that > > > > > converts > > > > > >>> perf.data file into gcov format that can be consumed by gcc > with > > > > > >>> -fauto-profile ( > > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > > > > > > > > > > > > I noticed that the source code for create_gcov has been > deleted > > > from > > > > > >>> https://github.com/google/autofdo on April 7. I asked about > that > > > > > change > > > > > >>> in that repo and got the following reply: > > > > > > > > > > > > > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738 > > > > > > > > > > > > "Actually we didn't use create_gcov and havn't updated > > > create_gcov > > > > > for > > > > > >>> years, and we also didn't have enough tests to guarantee it > works > > > (It > > > > > was > > > > > >>> gcc-4.8 when we used and verified create_gcov). If you need > it, it > > > is > > > > > >>> welcomed to update create_gcov and add it to the respository." > > > > > > > > > > > > Does this mean that AutoFDO is currently dead in gcc? > > > > > > > > > > Hello. > > > > > > > > > > Yes. I know that even basic test cases have been broken for > years > > > in > > > > > the > > > > > >>> GCC. > > > > > It's new to me that create_gcov was removed. > > > > > > > > > > I tend to send patch to GCC that will remove AutoFDO from GCC. > > > > > I known Bin spent some time working on AutoFDO, has he came > up to > > > > > >>> something? > > > > > >>> > > > > > >>> The GCC side of auto-FDO is not that hard. We have most of > > > > > >>> infrastructure in place, but stopping point for me was always > > > > > difficulty > > > > > >>> to get gcov-tool working. If some maintainer steps up, I > think I > > > can > > > > > >>> fix GCC side. > > > > > >>> > > > > > >>> I am bit unsure how important feature it is - we have FDO that > > > works > > > > > >>> quite well for most users but I know there are some users of > the > > > LLVM > > > > > >>> implementation and there is potential to tie this with other > > > hardware > > > > > >>> events to asist i.e. if conversion (where one wants to know how > > > well > > > > > CPU > > > > > >>> predicts the jump rather than just the jump probability) which > I > > > always > > > > > >>> found potentially interesting. > > > > > >>> > > > > > >>> Honza > > > > > > > > > > Martin > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Eugene > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >
Re: State of AutoFDO in GCC
On Fri, Apr 23, 2021 at 9:54 AM Jan Hubicka wrote: > > On Fri, Apr 23, 2021 at 12:18 AM Martin Liška wrote: > > > > > On 4/23/21 9:00 AM, Richard Biener via Gcc wrote: > > > > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc > > > > wrote: > > > >> > > > >> Hi, the create_gcov tool was probably removed with the assumption > that > > > it > > > >> was only used with Google GCC branch, but it is actually used with > GCC > > > >> trunk as well. > > > >> > > > >> Given that, the tool will be restored in the github repo. It seems > to > > > build > > > >> and work fine with the regression test. > > > >> > > > >> The tool may ust work as it is right now, but there is no guarantee > it > > > >> won't break in the future unless someone in the GCC community tries > to > > > >> maintain it. > > > > > > Hi. > > > > > > The current situation is that AutoFDO doesn't work with pretty simple > > > test-cases > > > we have in testsuite: > > > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672 > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379 > > > > > > These are ~5 years old and nothing has happened. > > Basic functionality is to identify hot parts of programs and give basic > idea of branch probabilities. Bugs above are about other optimizations > (indirect inlining, hot-cold partitioning and peeling). First one ought > to be working but still indirect call profiling accounts for relatively > minor portion of FDO benefits, so auto-FDO should be usefable even w/o > these working if tooling was fixed. Expecting hot-cold partitioning and > peeling to work reliably for testcases designed for normalFDO is IMO bit > unrealistic. > > Perf is not limited to x86, so we should get basic usability for all > major architectures. David, what CPUs are supported with auto-FDO on > LLVM side? > > > > This is a major feature in Clang. Deprecating it in GCC will be > unfortunate. > > I also like overall idea of auto-FDO just never had much time to get it > into shape. Every stage1 I am trying to fix something that was broken > for a while (like ICF last stage1) and fix it, so option would be to > focus on autoFDO this stage1 even though I have quite few other things > in TODO list. I never looked into the tools translating perf data to > gcc compatible format and how easy would be to keep this alive. > > David, how does perf->LLVM path work these days? > It uses create_llvm_prof tool which is in the same git repo. The data parsing part is shared with create_gcov, but the writer is obviously different for the two tools. David > Honza > > > > David > > > > > > > > Thoughts? > > > Martin > > > > > > > > > > > Having the tool third-party makes keeping the whole chain working > more > > > > difficult. > > > > > > > > Richard. > > > > > > > >> Thanks, > > > >> > > > >> David > > > >> > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka wrote: > > > >> > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool that > > > converts > > > >>> perf.data file into gcov format that can be consumed by gcc with > > > >>> -fauto-profile ( > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > > > >>>>> > > > >>>>> I noticed that the source code for create_gcov has been deleted > from > > > >>> https://github.com/google/autofdo on April 7. I asked about that > > > change > > > >>> in that repo and got the following reply: > > > >>>>> > > > >>>>> > https://github.com/google/autofdo/pull/107#issuecomment-819108738 > > > >>>>> > > > >>>>> "Actually we didn't use create_gcov and havn't updated > create_gcov > > > for > > > >>> years, and we also didn't have enough tests to guarantee it works > (It > > > was > > > >>> gcc-4.8 when we used and verified create_gcov). If you need it, it > is > > > >>> welcomed to update create_gcov and add it to the respositor
Re: State of AutoFDO in GCC
On Fri, Apr 23, 2021 at 12:18 AM Martin Liška wrote: > On 4/23/21 9:00 AM, Richard Biener via Gcc wrote: > > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc > > wrote: > >> > >> Hi, the create_gcov tool was probably removed with the assumption that > it > >> was only used with Google GCC branch, but it is actually used with GCC > >> trunk as well. > >> > >> Given that, the tool will be restored in the github repo. It seems to > build > >> and work fine with the regression test. > >> > >> The tool may ust work as it is right now, but there is no guarantee it > >> won't break in the future unless someone in the GCC community tries to > >> maintain it. > > Hi. > > The current situation is that AutoFDO doesn't work with pretty simple > test-cases > we have in testsuite: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672 > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379 > > These are ~5 years old and nothing has happened. > > I'm pretty sure the current autofdo can't emit a .gcda file format that > I've changed in the recent years. > I have not not looked into the details, but is it possible the problem is at GCC side (e.g, debug info quality etc)? > > > > > I think if we want to keep the feature it makes sense to provide > create_gcov > > functionality either directly from perf (input data producer) or from gcc > > (data consumer). Of course I have no idea about its complexity, license > > or implementation language ... > > For me, it's just an i386 feature (maybe aarch64 has perf counters too?), > supported > only by vendor (Intel) and I'm not planning working on that. > I don't like having a feature that is obviously broken and potential GCC > users get > bad experience every time they try to use it. > It must be working at sometime, so perhaps a bisect of GCC revisions can reveal when it regressed. > Can we at least deprecate the feature for GCC 11? If these is enough > interest, > we can fix it, if not, I would remove it in GCC 13 timeframe. > This is a major feature in Clang. Deprecating it in GCC will be unfortunate. David > > Thoughts? > Martin > > > > > Having the tool third-party makes keeping the whole chain working more > > difficult. > > > > Richard. > > > >> Thanks, > >> > >> David > >> > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka wrote: > >> > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > >>>>> GCC documentation for AutoFDO points to create_gcov tool that > converts > >>> perf.data file into gcov format that can be consumed by gcc with > >>> -fauto-profile ( > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > >>>>> > >>>>> I noticed that the source code for create_gcov has been deleted from > >>> https://github.com/google/autofdo on April 7. I asked about that > change > >>> in that repo and got the following reply: > >>>>> > >>>>> https://github.com/google/autofdo/pull/107#issuecomment-819108738 > >>>>> > >>>>> "Actually we didn't use create_gcov and havn't updated create_gcov > for > >>> years, and we also didn't have enough tests to guarantee it works (It > was > >>> gcc-4.8 when we used and verified create_gcov). If you need it, it is > >>> welcomed to update create_gcov and add it to the respository." > >>>>> > >>>>> Does this mean that AutoFDO is currently dead in gcc? > >>>> > >>>> Hello. > >>>> > >>>> Yes. I know that even basic test cases have been broken for years in > the > >>> GCC. > >>>> It's new to me that create_gcov was removed. > >>>> > >>>> I tend to send patch to GCC that will remove AutoFDO from GCC. > >>>> I known Bin spent some time working on AutoFDO, has he came up to > >>> something? > >>> > >>> The GCC side of auto-FDO is not that hard. We have most of > >>> infrastructure in place, but stopping point for me was always > difficulty > >>> to get gcov-tool working. If some maintainer steps up, I think I can > >>> fix GCC side. > >>> > >>> I am bit unsure how important feature it is - we have FDO that works > >>> quite well for most users but I know there are some users of the LLVM > >>> implementation and there is potential to tie this with other hardware > >>> events to asist i.e. if conversion (where one wants to know how well > CPU > >>> predicts the jump rather than just the jump probability) which I always > >>> found potentially interesting. > >>> > >>> Honza > >>>> > >>>> Martin > >>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Eugene > >>>>> > >>>> > >>> > >
Re: State of AutoFDO in GCC
On Fri, Apr 23, 2021 at 12:00 AM Richard Biener wrote: > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc > wrote: > > > > Hi, the create_gcov tool was probably removed with the assumption that it > > was only used with Google GCC branch, but it is actually used with GCC > > trunk as well. > > > > Given that, the tool will be restored in the github repo. It seems to > build > > and work fine with the regression test. > > > > The tool may ust work as it is right now, but there is no guarantee it > > won't break in the future unless someone in the GCC community tries to > > maintain it. > > I think if we want to keep the feature it makes sense to provide > create_gcov > functionality either directly from perf (input data producer) or from gcc > (data consumer). Of course I have no idea about its complexity, license > or implementation language ... > > Right. What it takes is a perf data (can be text format) parser to produce the format GCC needs, but someone in the community needs to take the lead. It should not involve too much effort. David Having the tool third-party makes keeping the whole chain working more > difficult. > > Richard. > > > Thanks, > > > > David > > > > On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka wrote: > > > > > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > > > > > GCC documentation for AutoFDO points to create_gcov tool that > converts > > > perf.data file into gcov format that can be consumed by gcc with > > > -fauto-profile ( > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > > > https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > > > > > > > > > > I noticed that the source code for create_gcov has been deleted > from > > > https://github.com/google/autofdo on April 7. I asked about that > change > > > in that repo and got the following reply: > > > > > > > > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738 > > > > > > > > > > "Actually we didn't use create_gcov and havn't updated create_gcov > for > > > years, and we also didn't have enough tests to guarantee it works (It > was > > > gcc-4.8 when we used and verified create_gcov). If you need it, it is > > > welcomed to update create_gcov and add it to the respository." > > > > > > > > > > Does this mean that AutoFDO is currently dead in gcc? > > > > > > > > Hello. > > > > > > > > Yes. I know that even basic test cases have been broken for years in > the > > > GCC. > > > > It's new to me that create_gcov was removed. > > > > > > > > I tend to send patch to GCC that will remove AutoFDO from GCC. > > > > I known Bin spent some time working on AutoFDO, has he came up to > > > something? > > > > > > The GCC side of auto-FDO is not that hard. We have most of > > > infrastructure in place, but stopping point for me was always > difficulty > > > to get gcov-tool working. If some maintainer steps up, I think I can > > > fix GCC side. > > > > > > I am bit unsure how important feature it is - we have FDO that works > > > quite well for most users but I know there are some users of the LLVM > > > implementation and there is potential to tie this with other hardware > > > events to asist i.e. if conversion (where one wants to know how well > CPU > > > predicts the jump rather than just the jump probability) which I always > > > found potentially interesting. > > > > > > Honza > > > > > > > > Martin > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Eugene > > > > > > > > > > > > >
Re: State of AutoFDO in GCC
Hi, the create_gcov tool was probably removed with the assumption that it was only used with Google GCC branch, but it is actually used with GCC trunk as well. Given that, the tool will be restored in the github repo. It seems to build and work fine with the regression test. The tool may ust work as it is right now, but there is no guarantee it won't break in the future unless someone in the GCC community tries to maintain it. Thanks, David On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka wrote: > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote: > > > GCC documentation for AutoFDO points to create_gcov tool that converts > perf.data file into gcov format that can be consumed by gcc with > -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, > https://gcc.gnu.org/wiki/AutoFDO/Tutorial). > > > > > > I noticed that the source code for create_gcov has been deleted from > https://github.com/google/autofdo on April 7. I asked about that change > in that repo and got the following reply: > > > > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738 > > > > > > "Actually we didn't use create_gcov and havn't updated create_gcov for > years, and we also didn't have enough tests to guarantee it works (It was > gcc-4.8 when we used and verified create_gcov). If you need it, it is > welcomed to update create_gcov and add it to the respository." > > > > > > Does this mean that AutoFDO is currently dead in gcc? > > > > Hello. > > > > Yes. I know that even basic test cases have been broken for years in the > GCC. > > It's new to me that create_gcov was removed. > > > > I tend to send patch to GCC that will remove AutoFDO from GCC. > > I known Bin spent some time working on AutoFDO, has he came up to > something? > > The GCC side of auto-FDO is not that hard. We have most of > infrastructure in place, but stopping point for me was always difficulty > to get gcov-tool working. If some maintainer steps up, I think I can > fix GCC side. > > I am bit unsure how important feature it is - we have FDO that works > quite well for most users but I know there are some users of the LLVM > implementation and there is potential to tie this with other hardware > events to asist i.e. if conversion (where one wants to know how well CPU > predicts the jump rather than just the jump probability) which I always > found potentially interesting. > > Honza > > > > Martin > > > > > > > > Thanks, > > > > > > Eugene > > > > > >
