AutoFDO tools for GCC

[Eugene Rozenfeld via Gcc]

Hello,

I've been the AutoFDO maintainer for the last 1.5 years. I've resurrected 
autoprofiledbootstrap build and made a number of other fixes/improvements 
(e.g., discriminator support).

The tools for AutoFDO (create_gcov, etc.) currently live in 
https://github.com/google/AutoFDO  repo and GCC AutoFDO documentation points 
users to that repo. That repo also has tools for LLVM AutoFDO.
https://github.com/google/AutoFDO  has several submodules: 
https://github.com/google/autofdo/blob/master/.gitmodules

I got a message from Snehasish (cc'd) that google intends to migrate the tools 
for LLVM to the LLVM repo and wants to archive 
https://github.com/google/AutoFDO. That will be a problem for AutoFDO in GCC. 
The idea to find a different home for GCC AutoFDO tools was discussed before on 
this alias but this becomes more urgent now. One idea was to build these tools 
from GCC repo and another was to produce gcov from perf tool directly. Andi 
(cc'd)  had some early unfinished prototype for latter.

Please let me know if you have thoughts on how we should proceed.

Thanks,

Eugene

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Hi Xionghu,

Yes, I'm pretty sure both of the fixes you mentioned are applicable to GCC 10. 
I'm not sure what the bar is for backporting fixes.
Jan, can you comment on that?
The fixes that Xionghu mentioned are:

Fix indirect call inlining with AutoFDO

https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0
AutoFDO: Don't try to promote indirect calls that result in recursive 
direct calls

https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ba125745d9e9fe90a18a2af8701b3269c5fdd468

Thanks,

Eugene

-Original Message-
From: Xionghu Luo  
Sent: Tuesday, July 26, 2022 6:41 PM
To: Eugene Rozenfeld ; Andi Kleen 
; Joseph Myers ; Jan Hubicka 
; gcc 
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

[You don't often get email from yinyuefen...@gmail.com. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

On 2022/7/27 09:31, Xionghu Luo wrote:
>
>
> On 2022/7/27 04:12, Eugene Rozenfeld via Gcc wrote:
>> Hello GCC community.
>>
>> I started this thread on the state of AutoFDO in GCC more than a year 
>> ago. Here is the first message in the thread:
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc
>> .gnu.org%2Fpipermail%2Fgcc%2F2021-April%2F235860.html&data=05%7C0
>> 1%7CEugene.Rozenfeld%40microsoft.com%7Ccda1f6374c584732950e08da6f7126
>> 83%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944829019804556%7C
>> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t%2B94vOYKb8SDZ9i86Fnin
>> iYXt7LTGXqbrbftH1TVqEs%3D&reserved=0
>>
>> Since then I committed a number of patches to revive AutoFDO in GCC:
>>
>> Fix a typo in an AutoFDO error
>> string<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D23691ddd3aa
>> 3ffe55892b2bff54f9a15a89de2b4&data=05%7C01%7CEugene.Rozenfeld%40m
>> icrosoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91a
>> b2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8ey
>> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C30
>> 00%7C%7C%7C&sdata=4WD%2FgKxeXEFz7V6pcqDsYxvAbSOP3ldxXHb%2FoMxV5kc
>> %3D&reserved=0>
>>
>> Update gen_autofdo_event.py and
>> gcc-auto-profile.<https://nam06.safelinks.protection.outlook.com/?url
>> =https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D
>> 01d402c5e0ac1ddf5618bbe316b50067625fda46&data=05%7C01%7CEugene.Ro
>> zenfeld%40microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf
>> 86f141af91ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWF
>> pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>> Mn0%3D%7C3000%7C%7C%7C&sdata=TA%2FhV%2FKsRF3MUZmUL%2FpTWbESdx03VK
>> PIudXrrBjAMvQ%3D&reserved=0>
>>
>> Fixes for AutoFDO
>> tests<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
>> %2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Df9ad3d5339fa
>> aaed6e15a7b27d90fbc66eb72f37&data=05%7C01%7CEugene.Rozenfeld%40mi
>> crosoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91ab
>> 2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8eyJ
>> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300
>> 0%7C%7C%7C&sdata=AvWCTytUxlcXq%2BpjkWJmhkmV0nH%2Fn0CzC4alU9XA9%2F
>> 4%3D&reserved=0>
>>
>> Fix indir-call-prof-2.c with
>> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
>> 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D0ed093c7c3
>> f755bc1cd80e5186abeb2f5c50ee0c&data=05%7C01%7CEugene.Rozenfeld%40
>> microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91
>> ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8e
>> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
>> 000%7C%7C%7C&sdata=3dlTtTfe4XOOm6BEy5YLWG0l3WlQdfbCyFiXs3Q7W1I%3D
>> &reserved=0>
>>
>> Fixes for AutoFDO
>> testing<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
>> 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D9265b37853
>> 1391498ec1727f67a45da72a6c07e9&data=05%7C01%7CEugene.Rozenfeld%40
>> microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91
>> ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8e
>> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
>> 000%7C%7C%7C&sdata=ZWSIQ0jb6t2DZQpDb%2F7e5FqKM6KKskM%2FAYzLpxbUkp
>> 4%3D&

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Hi Jan,

Thank you for your reply. I did have you on the TO line in my latest patch:
https://gcc.gnu.org/pipermail/gcc-patches/2022-June/596065.html

That's the patch I need a review on.

I'm looking forward to co-maintaining AutoFDO with you.

Thanks,

Eugene

-Original Message-
From: Jan Hubicka  
Sent: Wednesday, July 27, 2022 12:27 AM
To: David Edelsohn 
Cc: Eugene Rozenfeld ; Martin Liska 
; Xinliang David Li ; gcc 
; Andi Kleen ; Joseph Myers 

Subject: [EXTERNAL] Re: State of AutoFDO in GCC

> On Tue, Jul 26, 2022 at 4:13 PM Eugene Rozenfeld via Gcc 
>  wrote:
> >
> > Hello GCC community.
> >
> > I started this thread on the state of AutoFDO in GCC more than a 
> > year ago. Here is the first message in the thread: 
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgc
> > c.gnu.org%2Fpipermail%2Fgcc%2F2021-April%2F235860.html&data=05%7
> > C01%7CEugene.Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa
> > 1619e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63794503618637077
> > 0%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi
> > I6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9%2Bnv2ShWxKh88K%
> > 2BsOeqPgQX3lOCJQ0lnF%2F7SUs4K4uI%3D&reserved=0
> >
> > Since then I committed a number of patches to revive AutoFDO in GCC:
> >
> > Fix a typo in an AutoFDO error 
> > string<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
> > 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D23691ddd3
> > aa3ffe55892b2bff54f9a15a89de2b4&data=05%7C01%7CEugene.Rozenfeld%
> > 40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141a
> > f91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZsb
> > 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> > D%7C3000%7C%7C%7C&sdata=qkecfE9uH5gy91vILQQlCk9RpExqPZxO4q02wiN1
> > EFw%3D&reserved=0> Update gen_autofdo_event.py and 
> > gcc-auto-profile.<https://nam06.safelinks.protection.outlook.com/?ur
> > l=https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%
> > 3D01d402c5e0ac1ddf5618bbe316b50067625fda46&data=05%7C01%7CEugene
> > .Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f9
> > 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%
> > 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=E52qVFlfdfFGnW9yDsBNhh4k2ey8g
> > 3aJEGzH40MuSOc%3D&reserved=0> Fixes for AutoFDO 
> > tests<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Df9ad3d5339
> > faaaed6e15a7b27d90fbc66eb72f37&data=05%7C01%7CEugene.Rozenfeld%4
> > 0microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af
> > 91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZsb3
> > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C3000%7C%7C%7C&sdata=XYlFoY3OTTXHp18O1v8BY47A17NyNPXvUWWYsVnbD
> > 0U%3D&reserved=0> Fix indir-call-prof-2.c with 
> > AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> > %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D0ed093c7
> > c3f755bc1cd80e5186abeb2f5c50ee0c&data=05%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141
> > af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > 3D%7C3000%7C%7C%7C&sdata=sLftY6hjvzSuE9ZgkGmXZLDpRMjlDo%2FEAyDyP
> > CviY5Q%3D&reserved=0> Fixes for AutoFDO 
> > testing<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> > %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D9265b378
> > 531391498ec1727f67a45da72a6c07e9&data=05%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141
> > af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > 3D%7C3000%7C%7C%7C&sdata=lXZ%2F%2FbcfYD%2BQyIiXMAaCxOujEAfDXSY1p
> > 78kUb2md7w%3D&reserved=0> Fix indirect call inlining with 
> > AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> > %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D285aa689
> > 5d479bed8e72ad363290846645b6faa0&data=05%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141
> > af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZs
&

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Hi David,

Thank you for your offer to take on the responsibility of maintaining the 
AutoFDO-specific portions of the code. I'll be happy to do that with Jan's and 
other GGC maintainers' help.
I hope more people will start using and contributing to AutoFDO.

Thanks,

Eugene

-Original Message-
From: David Edelsohn  
Sent: Tuesday, July 26, 2022 3:38 PM
To: Eugene Rozenfeld ; Jan Hubicka 
; Martin Liska ; Xinliang David Li 

Cc: Andi Kleen ; Joseph Myers ; 
gcc 
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

[You don't often get email from dje@gmail.com. Learn why this is important 
at https://aka.ms/LearnAboutSenderIdentification ]

On Tue, Jul 26, 2022 at 4:13 PM Eugene Rozenfeld via Gcc  
wrote:
>
> Hello GCC community.
>
> I started this thread on the state of AutoFDO in GCC more than a year 
> ago. Here is the first message in the thread: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fpipermail%2Fgcc%2F2021-April%2F235860.html&data=05%7C01%
> 7CEugene.Rozenfeld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944718882043585%7CUnkn
> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwi
> LCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5RoiLypTlkNRIZW8yAufYR4qiO573
> 0PADO%2BFsS6InIU%3D&reserved=0
>
> Since then I committed a number of patches to revive AutoFDO in GCC:
>
> Fix a typo in an AutoFDO error 
> string<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D23691ddd3aa3f
> fe55892b2bff54f9a15a89de2b4&data=05%7C01%7CEugene.Rozenfeld%40micr
> osoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> 7C%7C&sdata=LRgW4qyh%2FAdrlKluUXnaDTFCNW8tnS1WX8bCs1WAT7s%3D&r
> eserved=0> Update gen_autofdo_event.py and 
> gcc-auto-profile.<https://nam06.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D01
> d402c5e0ac1ddf5618bbe316b50067625fda46&data=05%7C01%7CEugene.Rozen
> feld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f1
> 41af91ab2d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C&sdata=oBJvTW6RrkY6uURmr0UK5gaNiVS1b8vFfIyOoqq8AkM
> %3D&reserved=0> Fixes for AutoFDO 
> tests<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> 2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Df9ad3d5339faaa
> ed6e15a7b27d90fbc66eb72f37&data=05%7C01%7CEugene.Rozenfeld%40micro
> soft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> C%7C&sdata=6evlP10%2BQJyUuVZ3u%2Fv%2FP9nSIsASLjtWETyqeBfnQ2k%3D&am
> p;reserved=0> Fix indir-call-prof-2.c with 
> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D0ed093c7c3f7
> 55bc1cd80e5186abeb2f5c50ee0c&data=05%7C01%7CEugene.Rozenfeld%40mic
> rosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=mtMd21By6r8DiIIXzk5ePHP0iU%2FHfnDQG8%2FXJbAr5qE%3D&am
> p;reserved=0> Fixes for AutoFDO 
> testing<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D9265b3785313
> 91498ec1727f67a45da72a6c07e9&data=05%7C01%7CEugene.Rozenfeld%40mic
> rosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=QnKrxt7W9wfTYY3Drjm%2FSn7D4yHwxjInvI8dj9KEIs4%3D&
> reserved=0> Fix indirect call inlining with 
> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D285aa6895d47
> 9bed8e72ad363290846645b6faa0&data=05%7C01%7CEugene.Rozenfeld%40mic
> rosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=695ieVftlFRwICzxxMmmIb9%2F%2FDBlYBy9jJn%2ByfzkhP0

Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Hello GCC community.

I started this thread on the state of AutoFDO in GCC more than a year ago. Here 
is the first message in the thread: 
https://gcc.gnu.org/pipermail/gcc/2021-April/235860.html

Since then I committed a number of patches to revive AutoFDO in GCC:

Fix a typo in an AutoFDO error 
string
Update gen_autofdo_event.py and 
gcc-auto-profile.
Fixes for AutoFDO 
tests
Fix indir-call-prof-2.c with 
AutoFDO
Fixes for AutoFDO 
testing
Fix indirect call inlining with 
AutoFDO
Improve AutoFDO count propagation 
algorithm
AutoFDO: don't set param_early_inliner_max_iterations to 
10.
AutoFDO: Don't try to promote indirect calls that result in recursive direct 
calls
Fix profile count maintenance in vectorizer 
peeling.

I also made a number of fixes and improvements to create_gcov tool in 
https://github.com/google/autofdo .

AutoFDO in GCC is in a much better shape now.

I have a further set of patches that improve DWARF discriminator support in GCC 
and enable AutoFDO to use discriminators. It's based on commits in an old 
Google vendor branch as described in Andi's mail below
but uses a different approach for keeping track of per-instruction 
discriminators.

I submitted the first (and the biggest) of these patches almost 2 months ago on 
June 2: 
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0
but only got a review from Andi 
(https://gcc.gnu.org/pipermail/gcc-patches/2022-June/596549.html) who is not 
allowed to approve patches for commit. I pinged gcc-patches twice with no 
success.

I would appreciate help in getting a review on this patch so that I can get it 
committed and submit patches that depend on it.

Thank you,

Eugene

-Original Message-
From: Andi Kleen 
Sent: Monday, May 10, 2021 10:21 AM
To: Joseph Myers 
Cc: Jan Hubicka ; gcc ; Eugene Rozenfeld 

Subject: [EXTERNAL] Re: State of AutoFDO in GCC

On Mon, May 10, 2021 at 04:55:50PM +, Joseph Myers wrote:
> On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
>
> > It's difficult to find now because it was a branch in the old SVN
> > that wasn't converted. Sadly the great git conversion was quite lossy.
>
> All branches and tags, including deleted ones, were converted (under
> not-fetched-by-default refs in some cases); the git repository has
> everything that might plausibly be useful, omitting only a few things
> that would have been meaningless to convert, such as mistaken branch
> creations in the root of the repository and the SVN hooks directory.
> Use "git ls-remote git://gcc.gnu.org/git/gcc.git" to see the full list
> of over 5000 refs available in the repository (or do a clone with
> --mirror to fetch them all).

Okay thanks. I don't see them in any of the web interfaces, neither on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fgit%2Fgitweb.cgi%3Fp%3Dgcc.git&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ts53XULDtR3o7fevlntCJdtzRqTo9R85LrxJ0ZfOBnE%3D&reserved=0
nor on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgcc-mirror%2Fgcc&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FPGF3vy3hD1OwiXmWzkUnOt9%2BR3YArZw0kCVueOKYpc%3D&reserved=0
but
git fetch origin vendors/google/heads/gcc-4_8 does the trick for fetching the 
commits, but not the symbolic branches.

Anyways with that it looks like the discriminator changes are:

commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
Author: Dehao Chen mailto:de...@gcc.gnu.org>>
Date:   Thu Oct 10 14:39:31 2013 +

Use only lineno+discriminator (remove the callee function name) as the key 
to represent callsite. Because each callsite will have its discriminator if in 
the same line.

2013-10-10  Dehao Chen  

RE: [EXTERNAL] Re: Reporting lto bugs

[Eugene Rozenfeld via Gcc]

My scenario is not cygwin. It's linux cross-compile x86_64 to aarch64 in an 
Ubuntu vm so I believe this should produce line info with -flto -g yet I see 
"No Line Number Statements" after Directory Table and File Name Table in the 
output of objdump -g.
DWARF Version is 4.
I verified that I get line numbers if I don't specify -flto.
I have all the lto1 commands and I'd like to try to debug this myself first. Do 
you have any tips on how to debug this?

Thanks,

Eugene

-Original Message-
From: Richard Biener  
Sent: Friday, September 24, 2021 12:36 AM
To: Eugene Rozenfeld 
Cc: gcc@gcc.gnu.org
Subject: [EXTERNAL] Re: Reporting lto bugs

On Fri, Sep 24, 2021 at 3:45 AM Eugene Rozenfeld via Gcc  
wrote:
>
> I ran into a bug with lto: no line number info gets emitted when building my 
> project with -flto and -g (with gcc 8.2). I'd like to provide a repro in the 
> bug report but I don't know if there is an easy way to collect everything.
> The instructions at 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fbugs%2F&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cb7b3ef43536943c30e3108d97f2df355%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637680657602091864%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yGG5QtNDKjD0z4Q%2FcNmuwx5CZV8cElqGetmfmYzAwjA%3D&reserved=0
>  seem to cover non-lto compilations.
> For MSVC there is an easy way to collect linker repro files for ltcg: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs
> .microsoft.com%2Fen-us%2Fcpp%2Foverview%2Fhow-to-report-a-problem-with
> -the-visual-cpp-toolset%3Fview%3Dmsvc-160%23link-repros&data=04%7C
> 01%7CEugene.Rozenfeld%40microsoft.com%7Cb7b3ef43536943c30e3108d97f2df3
> 55%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637680657602101859%7CU
> nknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%3D%7C1000&sdata=WhtAKrM1LoveH1Jbi3Ulhrp7hUdXJ46hgh1
> osT%2BB1nc%3D&reserved=0
> Is there something similar for GCC?

We generally prefer preprocessed sources here as with other bugs - for LTO that 
might mean including quite a bunch of files but there's instructions on the 
wiki on how to reduce LTO testcases:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fwiki%2FA_guide_to_testcase_reduction%23Reducing_LTO_bugs&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cb7b3ef43536943c30e3108d97f2df355%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637680657602101859%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GExhyUNJvSQgMPrSGccVj8woBTLahYVAjI%2FBvQ79QMo%3D&reserved=0

Note that for mingw and cygwin you likely run into the issue that LTO does not 
support -g there (sic), see a similar issue for darwin (PR82005), there must be 
a bug for mingw/cygwin as well but I can't find it right now.

Richard.

> Thanks,
>
> Eugene

Reporting lto bugs

[Eugene Rozenfeld via Gcc]

I ran into a bug with lto: no line number info gets emitted when building my 
project with -flto and -g (with gcc 8.2). I'd like to provide a repro in the 
bug report but I don't know if there is an easy way to collect everything.
The instructions at https://gcc.gnu.org/bugs/ seem to cover non-lto 
compilations.
For MSVC there is an easy way to collect linker repro files for ltcg: 
https://docs.microsoft.com/en-us/cpp/overview/how-to-report-a-problem-with-the-visual-cpp-toolset?view=msvc-160#link-repros
Is there something similar for GCC?

Thanks,

Eugene

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Hi Andy,

I'm trying to revive autofdo testing. One of the issues I'm running into with 
my setup is that PEBS doesn't work for with perf record even though PEBS is 
enabled.
I'm running Ubuntu 20.04 in a Hyper-V virtual machine; the processor is Icelake 
(GenuineIntel-6-7E).

I did the following:

1. Enabled pmu, lbr, and pebs in my Hyper-V virtual machine as described in 
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/performance-monitoring-hardware
2. Verified that pmu, lbr, and pebs are enabled in the vm by running 
erozen@erozen-Virtual-Machine:~/objdir/gcc$ dmesg | egrep -i 'pmu'
[0.266474] Performance Events: PEBS fmt4+, Icelake events, 32-deep 
LBR, full-width counters, Intel PMU driver.
3. Ran
erozen@erozen-Virtual-Machine:~/objdir/gcc$ perf record -e 
cpu/event=0xc4,umask=0x20/pu -b -m8 true -v
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) 
for event (cpu/event=0xc4,umask=0x20/pu).
/bin/dmesg | grep -i perf may provide additional information.

Omitting /p works fine:
erozen@erozen-Virtual-Machine:~/objdir/gcc$ perf record -e 
cpu/event=0xc4,umask=0x20/u -b -m8 true -v
[ perf record: Woken up 0 times to write data ]
[ perf record: Captured and wrote 0.007 MB perf.data (11 samples) ]

Is there a way to get PEBS working with perf record in a vm? I would appreciate 
any pointers on how to investigate this.

The version of perf I'm using is 5.8.18.

Thanks,

Eugene

-Original Message-
From: Andi Kleen  
Sent: Friday, April 30, 2021 2:46 PM
To: Eugene Rozenfeld via Gcc 
Cc: Xinliang David Li ; Richard Biener 
; Eugene Rozenfeld 
; Jan Hubicka 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

Eugene Rozenfeld via Gcc  writes:

> Is the format produced by create_gcov and expected by GCC under 
> -fauto-rpofile documented somewhere? How is it different from .gcda 
> used in FDO, e.g., as described here:
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsrc.gnu-darwin.org%2Fsrc%2Fcontrib%2Fgcc%2Fgcov-io.h.html&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C6c14ea3d93c44364845008d90c214cb6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637554159427749575%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=q9ROiOTma41UeQ%2FQG%2BUktEOrHAWonTTpcPRPmx%2Fgw0g%3D&reserved=0?

I believe it's very similar.

> I would prefer that AutoFDO is not removed from GCC and it would be 
> helpful if create_gcov were restored in google/autofdo. I checked out 
> a revision before the recent merge and tried it on a simple example 
> and it seems to work.
> I'm also interested in contributing improvements for AutoFDO so will 
> try to investigate 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D71672&data=04%7C01%7CEuge
> ne.Rozenfeld%40microsoft.com%7C6c14ea3d93c44364845008d90c214cb6%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637554159427749575%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn0%3D%7C1000&sdata=99Igueuxq7AoHU%2B20BZs4E4K5rgdPFCiR8eygKaJdK
> E%3D&reserved=0 and 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D81379&data=04%7C01%7CEuge
> ne.Rozenfeld%40microsoft.com%7C6c14ea3d93c44364845008d90c214cb6%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637554159427759566%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn0%3D%7C1000&sdata=Es91Dtt5Wt6%2BJtPWxHhkHqdWBVwzCiF5PcuXoHjY%2
> Bzs%3D&reserved=0

That would be great.

-Andi

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

It appears that create_gcov doesn't support binaries with dwarf version 5 
(which is the current default). 
I tried a trivial example and got reasonable gcov files for binaries with dwarf 
v2, v3, and v4 but the same example with dwarf v5 produced 
"File './sort' has mangled .debug_info section."
and a gcov file with 0 functions.

Does create_llvm_prof has the same limitation?

Eugene

-Original Message-
From: Wei Mi  
Sent: Wednesday, May 26, 2021 7:52 PM
To: Eugene Rozenfeld 
Cc: Andi Kleen ; Hongtao Yu ; Xinliang David 
Li ; Jan Hubicka ; gcc@gcc.gnu.org; Wenlei 
He 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

Thanks. Good to know the build works with newer protobuf.

Wei.

On Wed, May 26, 2021 at 4:40 PM Eugene Rozenfeld 
 wrote:
>
> 3.0.0 is the latest supported version on Ubuntu 18.04. I verified that the 
> build works on Ubuntu 20.04 with  protobuf-compiler and libprotobuf-dev 
> version 3.6.1.3.
>
> Eugene
>
> -Original Message-
> From: Wei Mi 
> Sent: Tuesday, May 25, 2021 8:07 PM
> To: Eugene Rozenfeld 
> Cc: Andi Kleen ; Hongtao Yu ; Xinliang 
> David Li ; Jan Hubicka ; 
> gcc@gcc.gnu.org; Wenlei He 
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> I checked the source of protobuf 3.0.0 and it didn't contain the operator[] 
> in RepeatedField. Need to install a newer version of protobuf.
>
> Thanks,
> Wei.
>
> On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld 
>  wrote:
> >
> > Both are 3.0.0-9.1ubuntu1:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> > protobuf-compiler Listing... Done protobuf-compiler/bionic,now
> > 3.0.0-9.1ubuntu1 amd64 [installed]
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> > libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now
> > 3.0.0-9.1ubuntu1 amd64 [installed]
> >
> > -Original Message-
> > From: Wei Mi 
> > Sent: Tuesday, May 25, 2021 9:17 AM
> > To: Eugene Rozenfeld 
> > Cc: Andi Kleen ; Hongtao Yu ; 
> > Xinliang David Li ; Jan Hubicka 
> > ; gcc@gcc.gnu.org; Wenlei He 
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It looks like some version problem about protobuf-compiler and 
> > libprotobuf-dev. Could you check what is the installed version on your end 
> > for those two packages and see if they are consistent?
> >
> > On my platform, they are both 3.12.4.
> >
> > On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld 
> >  wrote:
> > >
> > > That eliminates the previous error but there is a new one:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] 
> > > Building CXX object 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > ip
> > > pe
> > > r/perf_reader.cc.o
> > > FAILED: 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > > /usr/bin/c++   -I../third_party/perf_data_converter/src 
> > > -I../third_party/perf_data_converter/src/quipper -I../ 
> > > -I../third_party/glog/src -I../third_party/abseil -I../util -I. 
> > > -Ithird_party/glog -std=gnu++1z -MD -MT 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > >  -MF 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d
> > >  -o 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > >  -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member 
> > > function 'bool 
> > > quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, 
> > > size_t)':
> > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: 
> > > error: no match for 'operator[]' (operand types are 'const 
> > > google::protobuf::RepeatedField' and 'int')
> > >  nrcpus = proto_uint32_metadata.data()[0];
> > >
> > > -Original Message-
> > > From: Wei Mi 
> > > Sent: Monday, May 24, 2021 8:54 PM
> > > To: Eugene Rozenfeld 
> > > Cc: Andi Kleen ; Hongtao Yu ; 
> > > Xinliang David Li ; Jan Hubicka 
> > > ; gcc@gcc.gnu.org; Wenlei He 
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It isn't exposed on my platform either. Looks like a bug in 
> > > perf_data_converter (i.e., quipper). Could you try adding #include 
> > >  in 
> > > third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see 
> > > if it fixes the problem? If it works, I will need to file a bug against 
> > > perf_data_converter.
> > >
> > > Thanks,
> > > Wei.
> > >
> > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld 
> > >  wrote:
> > > >
> > > > That fixed the error I saw before but the build still fails. The 
> > > > errors start with
> > > >
> > > >
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > > >
> > > > [2/217] Building CXX object
> > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/
> > > > qu
> > > > ip
> > > > pe
> > > > r/huge_page_d

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

3.0.0 is the latest supported version on Ubuntu 18.04. I verified that the 
build works on Ubuntu 20.04 with  protobuf-compiler and libprotobuf-dev version 
3.6.1.3.

Eugene

-Original Message-
From: Wei Mi  
Sent: Tuesday, May 25, 2021 8:07 PM
To: Eugene Rozenfeld 
Cc: Andi Kleen ; Hongtao Yu ; Xinliang David 
Li ; Jan Hubicka ; gcc@gcc.gnu.org; Wenlei 
He 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

I checked the source of protobuf 3.0.0 and it didn't contain the operator[] in 
RepeatedField. Need to install a newer version of protobuf.

Thanks,
Wei.

On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld 
 wrote:
>
> Both are 3.0.0-9.1ubuntu1:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> protobuf-compiler Listing... Done protobuf-compiler/bionic,now 
> 3.0.0-9.1ubuntu1 amd64 [installed] 
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now 
> 3.0.0-9.1ubuntu1 amd64 [installed]
>
> -Original Message-
> From: Wei Mi 
> Sent: Tuesday, May 25, 2021 9:17 AM
> To: Eugene Rozenfeld 
> Cc: Andi Kleen ; Hongtao Yu ; Xinliang 
> David Li ; Jan Hubicka ; 
> gcc@gcc.gnu.org; Wenlei He 
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It looks like some version problem about protobuf-compiler and 
> libprotobuf-dev. Could you check what is the installed version on your end 
> for those two packages and see if they are consistent?
>
> On my platform, they are both 3.12.4.
>
> On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld 
>  wrote:
> >
> > That eliminates the previous error but there is a new one:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] 
> > Building CXX object 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > pe
> > r/perf_reader.cc.o
> > FAILED: 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > /usr/bin/c++   -I../third_party/perf_data_converter/src 
> > -I../third_party/perf_data_converter/src/quipper -I../ 
> > -I../third_party/glog/src -I../third_party/abseil -I../util -I. 
> > -Ithird_party/glog -std=gnu++1z -MD -MT 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> >  -MF 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d
> >  -o 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> >  -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member 
> > function 'bool 
> > quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: 
> > error: no match for 'operator[]' (operand types are 'const 
> > google::protobuf::RepeatedField' and 'int')
> >  nrcpus = proto_uint32_metadata.data()[0];
> >
> > -Original Message-
> > From: Wei Mi 
> > Sent: Monday, May 24, 2021 8:54 PM
> > To: Eugene Rozenfeld 
> > Cc: Andi Kleen ; Hongtao Yu ; 
> > Xinliang David Li ; Jan Hubicka 
> > ; gcc@gcc.gnu.org; Wenlei He 
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It isn't exposed on my platform either. Looks like a bug in 
> > perf_data_converter (i.e., quipper). Could you try adding #include 
> >  in 
> > third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if 
> > it fixes the problem? If it works, I will need to file a bug against 
> > perf_data_converter.
> >
> > Thanks,
> > Wei.
> >
> > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld 
> >  wrote:
> > >
> > > That fixed the error I saw before but the build still fails. The 
> > > errors start with
> > >
> > >
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > >
> > > [2/217] Building CXX object
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > ip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > FAILED:
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > ip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > /usr/bin/c++   -I../third_party/perf_data_converter/src 
> > > -I../third_party/perf_data_converter/src/quipper -I../ 
> > > -I../third_party/glog/src -I../third_party/abseil -I../util -I. 
> > > -Ithird_party/glog -std=gnu++1z -MD -MT 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
> > >  -MF 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d
> > >  -o 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
> > >  -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > >
> > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > 14
> > > 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> > > template type
> > >
> > >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Both are 3.0.0-9.1ubuntu1:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list protobuf-compiler
Listing... Done
protobuf-compiler/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list libprotobuf-dev
Listing... Done
libprotobuf-dev/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]

-Original Message-
From: Wei Mi  
Sent: Tuesday, May 25, 2021 9:17 AM
To: Eugene Rozenfeld 
Cc: Andi Kleen ; Hongtao Yu ; Xinliang David 
Li ; Jan Hubicka ; gcc@gcc.gnu.org; Wenlei 
He 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It looks like some version problem about protobuf-compiler and libprotobuf-dev. 
Could you check what is the installed version on your end for those two 
packages and see if they are consistent?

On my platform, they are both 3.12.4.

On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld 
 wrote:
>
> That eliminates the previous error but there is a new one:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] Building 
> CXX object 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> r/perf_reader.cc.o
> FAILED: 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> /usr/bin/c++   -I../third_party/perf_data_converter/src 
> -I../third_party/perf_data_converter/src/quipper -I../ 
> -I../third_party/glog/src -I../third_party/abseil -I../util -I. 
> -Ithird_party/glog -std=gnu++1z -MD -MT 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
>  -MF 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d
>  -o 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
>  -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member 
> function 'bool 
> quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: 
> no match for 'operator[]' (operand types are 'const 
> google::protobuf::RepeatedField' and 'int')
>  nrcpus = proto_uint32_metadata.data()[0];
>
> -Original Message-
> From: Wei Mi 
> Sent: Monday, May 24, 2021 8:54 PM
> To: Eugene Rozenfeld 
> Cc: Andi Kleen ; Hongtao Yu ; Xinliang 
> David Li ; Jan Hubicka ; 
> gcc@gcc.gnu.org; Wenlei He 
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It isn't exposed on my platform either. Looks like a bug in 
> perf_data_converter (i.e., quipper). Could you try adding #include 
>  in 
> third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if 
> it fixes the problem? If it works, I will need to file a bug against 
> perf_data_converter.
>
> Thanks,
> Wei.
>
> On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld 
>  wrote:
> >
> > That fixed the error I saw before but the build still fails. The 
> > errors start with
> >
> >
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> >
> > [2/217] Building CXX object
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > pe
> > r/huge_page_deducer.cc.o
> >
> > FAILED:
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > pe
> > r/huge_page_deducer.cc.o
> >
> > /usr/bin/c++   -I../third_party/perf_data_converter/src 
> > -I../third_party/perf_data_converter/src/quipper -I../ 
> > -I../third_party/glog/src -I../third_party/abseil -I../util -I. 
> > -Ithird_party/glog -std=gnu++1z -MD -MT 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
> >  -MF 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d
> >  -o 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
> >  -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> >
> > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > 14
> > 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> > template type
> >
> >using container = std::unordered_map;
> >
> >   ^
> >
> >
> >
> >
> >
> >
> >
> > From: Wei Mi 
> > Sent: Monday, May 24, 2021 8:12 PM
> > To: Eugene Rozenfeld 
> > Cc: Andi Kleen ; Hongtao Yu ; 
> > Xinliang David Li ; Jan Hubicka 
> > ; gcc@gcc.gnu.org; Wenlei He 
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> >
> >
> > Sorry, I added dependency for create_gcov but missed it for dump_gcov. 
> > Fixed it at 
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315397315%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9G

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

That eliminates the previous error but there is a new one:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[3/199] Building CXX object 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
FAILED: 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
 
/usr/bin/c++   -I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../ 
-I../third_party/glog/src -I../third_party/abseil -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
 -MF 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d
 -o 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
 -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member 
function 'bool 
quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: 
no match for 'operator[]' (operand types are 'const 
google::protobuf::RepeatedField' and 'int')
 nrcpus = proto_uint32_metadata.data()[0];

-Original Message-
From: Wei Mi  
Sent: Monday, May 24, 2021 8:54 PM
To: Eugene Rozenfeld 
Cc: Andi Kleen ; Hongtao Yu ; Xinliang David 
Li ; Jan Hubicka ; gcc@gcc.gnu.org; Wenlei 
He 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It isn't exposed on my platform either. Looks like a bug in perf_data_converter 
(i.e., quipper). Could you try adding #include  in 
third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it 
fixes the problem? If it works, I will need to file a bug against 
perf_data_converter.

Thanks,
Wei.

On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld 
 wrote:
>
> That fixed the error I saw before but the build still fails. The 
> errors start with
>
>
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
>
> [2/217] Building CXX object 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> r/huge_page_deducer.cc.o
>
> FAILED: 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> r/huge_page_deducer.cc.o
>
> /usr/bin/c++   -I../third_party/perf_data_converter/src 
> -I../third_party/perf_data_converter/src/quipper -I../ 
> -I../third_party/glog/src -I../third_party/abseil -I../util -I. 
> -Ithird_party/glog -std=gnu++1z -MD -MT 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
>  -MF 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d
>  -o 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
>  -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
>
> ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:14
> 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> template type
>
>using container = std::unordered_map;
>
>   ^
>
>
>
>
>
>
>
> From: Wei Mi 
> Sent: Monday, May 24, 2021 8:12 PM
> To: Eugene Rozenfeld 
> Cc: Andi Kleen ; Hongtao Yu ; Xinliang 
> David Li ; Jan Hubicka ; 
> gcc@gcc.gnu.org; Wenlei He 
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
>
>
> Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed 
> it at 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816277204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hwJ%2BG64Yw%2BEGMAV7kzilOjAOkHoOv3TQpdqkzcHGO%2FM%3D&reserved=0.
>
>
>
> Thanks,
>
> Wei.
>
>
>
> On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld 
>  wrote:
>
> Thank you Wei. Looks like something is still missing. This time 
> perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 
> with cmake 3.12.1):
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] Building 
> CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil 
> -I../third_party/perf_data_converter/src 
> -I../third_party/perf_data_converter/src/quipper -I../util -I. 
> -Ithird_party/glog -std=gnu++1z -MD -MT 
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF 
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o 
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> In file included from 
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>  from ../sample_reader.h:18,
>  from ../profile.h:15,
>   

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

That fixed the error I saw before but the build still fails. The errors start 
with

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[2/217] Building CXX object 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
FAILED: 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
/usr/bin/c++   -I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../ 
-I../third_party/glog/src -I../third_party/abseil -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
 -MF 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d
 -o 
CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
 -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:144:26: 
error: 'unordered_map' in namespace 'std' does not name a template type
   using container = std::unordered_map;
  ^



From: Wei Mi 
Sent: Monday, May 24, 2021 8:12 PM
To: Eugene Rozenfeld 
Cc: Andi Kleen ; Hongtao Yu ; Xinliang David 
Li ; Jan Hubicka ; gcc@gcc.gnu.org; Wenlei 
He 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it 
at 
https://github.com/google/autofdo/commit/6ca36cdc30986f13583a3aef3e27746ca4fc5bf6.

Thanks,
Wei.

On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld 
mailto:eugene.rozenf...@microsoft.com>> wrote:
Thank you Wei. Looks like something is still missing. This time perf_data.pb.h 
is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
/usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil 
-I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF 
CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o 
CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: 
"DISALLOW_COPY_AND_ASSIGN" redefined
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \

In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \

In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: 
"arraysize" redefined
 #define arraysize(x) (sizeof(x) / sizeof(*x))

In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
 #define arraysize(array) (sizeof(ArraySizeHelper(array)))

In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal 
error: perf_data.pb.h: No such file or directory
 #include "perf_data.pb.h"
  ^~~~
compilation terminated.
[6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
ninja: build stopped: subcommand failed.

Thanks,

Eugene

From: Wei Mi mailto:w...@google.com>>
Sent: Saturday, May 22, 2021 9:37 AM
To: Eugene Rozenfeld 
mailto:eugene.rozenf...@microsoft.com>>
Cc: Andi Kleen mailto:a...@linux.intel.com>>; Hongtao Yu 
mailto:h...@fb.com>>; Xinliang David Li 
mailto:davi...@google.com>>; Jan Hubicka 
mailto:hubi...@ucw.cz>>; 
gcc@gcc.gnu.org; Wenlei He 
mailto:wen...@fb.com>>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It is a p

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Thank you Wei. Looks like something is still missing. This time perf_data.pb.h 
is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o 
/usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil 
-I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF 
CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o 
CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: 
"DISALLOW_COPY_AND_ASSIGN" redefined
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
 
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
 
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: 
"arraysize" redefined
 #define arraysize(x) (sizeof(x) / sizeof(*x))
 
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
 #define arraysize(array) (sizeof(ArraySizeHelper(array)))
 
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal 
error: perf_data.pb.h: No such file or directory
 #include "perf_data.pb.h"
  ^~~~
compilation terminated.
[6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
ninja: build stopped: subcommand failed.

Thanks,

Eugene

From: Wei Mi  
Sent: Saturday, May 22, 2021 9:37 AM
To: Eugene Rozenfeld 
Cc: Andi Kleen ; Hongtao Yu ; Xinliang David 
Li ; Jan Hubicka ; gcc@gcc.gnu.org; Wenlei 
He 
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It is a proto library build dependency issue which didn't expose on my 
platform. I fix it at 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342270322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TxMivTXj9w0W5OFLunWcOpqf9wefrrUd0m3RqV8JLaA%3D&reserved=0.
 Please try again.

Thanks,
Wei.

On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld 
 wrote:
I tried following the instructions in "2.2 Build autofdo tool for gcc" in 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342280318%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KBPIcf5k1v230tbLke9%2FdSt25nbPrHhpBvPb8PiebPY%3D&reserved=0and
 got build failures:
 
eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o 
/usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil 
-I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF 
CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o 
CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
     from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: 
"DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from 
../thir

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

I tried following the instructions in "2.2 Build autofdo tool for gcc" in 
https://github.com/google/autofdo#readme
and got build failures:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
/usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil 
-I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF 
CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o 
CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: 
"DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: 
"arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal 
error: perf_stat.pb.h: No such file or directory
#include "perf_stat.pb.h"
  ^~~~
compilation terminated.

What is supposed to generate perf_stat.pb.h?

Thanks,

Eugene

From: Wei Mi 
Sent: Monday, May 10, 2021 4:47 PM
To: Andi Kleen 
Cc: Hongtao Yu ; Xinliang David Li ; Jan 
Hubicka ; gcc@gcc.gnu.org; Eugene Rozenfeld 
; Wenlei He 
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

https://github.com/google/autofdo
 has been updated. Now create_gcov/dump_gcov are added back and can be built 
separately.

Please look at "2.2 Build autofdo tool for gcc" in 
https://github.com/google/autofdo#readme

On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen 
mailto:a...@linux.intel.com>> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +, Hongtao Yu wrote:
> >Andi, thanks for pointing out the perf script issues. Can you please
> >elaborate a bit on the exact issue you have seen? We've been using
> >specific output of perf script such as mmap, LBR and callstack events
> >filtered by process id. It works fine so far but may certainly hit issues
> >in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://github.com/andikleen/autofdo

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

Is the format produced by create_gcov and expected by GCC under -fauto-rpofile 
documented somewhere? How is it different from .gcda used in FDO, e.g., as 
described here: http://src.gnu-darwin.org/src/contrib/gcc/gcov-io.h.html?
My input data is different from perf.data and I'd like to write a tool that 
produces the format needed for AutoFDO.

I would prefer that AutoFDO is not removed from GCC and it would be helpful if 
create_gcov were restored in google/autofdo. I checked out a revision before 
the recent merge and tried it on a simple example and it seems to work.
I'm also interested in contributing improvements for AutoFDO so will try to 
investigate https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672 and 
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379

Thanks,

Eugene

From: Xinliang David Li 
Sent: Friday, April 23, 2021 9:36 AM
To: Richard Biener 
Cc: Jan Hubicka ; gcc@gcc.gnu.org; Eugene Rozenfeld 

Subject: [EXTERNAL] Re: State of AutoFDO in GCC



On Fri, Apr 23, 2021 at 12:00 AM Richard Biener 
mailto:richard.guent...@gmail.com>> wrote:
On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
mailto:gcc@gcc.gnu.org>> wrote:
>
> Hi, the create_gcov tool was probably removed with the assumption that it
> was only used with Google GCC branch, but it is actually used with GCC
> trunk as well.
>
> Given that, the tool will be restored in the github repo. It seems to build
> and work fine with the regression test.
>
> The tool may ust work as it is right now, but there is no guarantee it
> won't break in the future unless someone in the GCC community tries to
> maintain it.

I think if we want to keep the feature it makes sense to provide create_gcov
functionality either directly from perf (input data producer) or from gcc
(data consumer).  Of course I have no idea about its complexity, license
or implementation language ...

Right. What it takes is a perf data (can be text format) parser to produce the 
format GCC needs, but someone in the community needs to take the lead. It 
should not involve too much effort.

David

Having the tool third-party makes keeping the whole chain working more
difficult.

Richard.

> Thanks,
>
> David
>
> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka 
> mailto:hubi...@ucw.cz>> wrote:
>
> > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > GCC documentation for AutoFDO points to create_gcov tool that converts
> > perf.data file into gcov format that can be consumed by gcc with
> > -fauto-profile 
> > (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fonlinedocs%2Fgcc%2FOptimize-Options.html&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772845557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FAYvyG0%2BxQ%2BbgLgEHUckUngTXDLoNJ4AASMeMVAhHWE%3D&reserved=0>,
> > https://gcc.gnu.org/wiki/AutoFDO/Tutorial<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fwiki%2FAutoFDO%2FTutorial&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63754792577285%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IqMDaxmMTM64eavwksTEXZKwlVhR5UZZiL4tfcyu3io%3D&reserved=0>).
> > > >
> > > > I noticed that the source code for create_gcov has been deleted from
> > https://github.com/google/autofdo<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772865552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6m86ZH8vmrXFIJOGO8WPYBxap1R4uULFZi5mE04dFbc%3D&reserved=0>
> >  on April 7. I asked about that change
> > in that repo and got the following reply:
> > > >
> > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fpull%2F107%23issuecomment-819108738&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772875543%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9k945Ma9HRo4RzafSMzr%2FiKx8ochLr77GyIO7pzBVJ8%3D&reserved=0>
> > > >
> > > > "Actually we didn't use create_gcov and havn't updated create_gcov f

State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

GCC documentation for AutoFDO points to create_gcov tool that converts 
perf.data file into gcov format that can be consumed by gcc with -fauto-profile 
(https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, 
https://gcc.gnu.org/wiki/AutoFDO/Tutorial).

I noticed that the source code for create_gcov has been deleted from 
https://github.com/google/autofdo on April 7. I asked about that change in that 
repo and got the following reply:

https://github.com/google/autofdo/pull/107#issuecomment-819108738

"Actually we didn't use create_gcov and havn't updated create_gcov for years, 
and we also didn't have enough tests to guarantee it works (It was gcc-4.8 when 
we used and verified create_gcov). If you need it, it is welcomed to update 
create_gcov and add it to the respository."

Does this mean that AutoFDO is currently dead in gcc?

Thanks,

Eugene