Yeah, create_llvm_prof depends on llvm_symbolizer which shouldn't have the same limitation. To separate from llvm, create_gcov still depends on the old set of addr2line library which likely doesn't support dwarf v5. That is an issue indeed.
Thanks, Wei. On Fri, Jun 11, 2021 at 6:14 PM Eugene Rozenfeld < eugene.rozenf...@microsoft.com> wrote: > It appears that create_gcov doesn't support binaries with dwarf version 5 > (which is the current default). > I tried a trivial example and got reasonable gcov files for binaries with > dwarf v2, v3, and v4 but the same example with dwarf v5 produced > "File './sort' has mangled .debug_info section." > and a gcov file with 0 functions. > > Does create_llvm_prof has the same limitation? > > Eugene > > -----Original Message----- > From: Wei Mi <w...@google.com> > Sent: Wednesday, May 26, 2021 7:52 PM > To: Eugene Rozenfeld <eugene.rozenf...@microsoft.com> > Cc: Andi Kleen <a...@linux.intel.com>; Hongtao Yu <h...@fb.com>; Xinliang > David Li <davi...@google.com>; Jan Hubicka <hubi...@ucw.cz>; > gcc@gcc.gnu.org; Wenlei He <wen...@fb.com> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > Thanks. Good to know the build works with newer protobuf. > > Wei. > > On Wed, May 26, 2021 at 4:40 PM Eugene Rozenfeld < > eugene.rozenf...@microsoft.com> wrote: > > > > 3.0.0 is the latest supported version on Ubuntu 18.04. I verified that > the build works on Ubuntu 20.04 with protobuf-compiler and libprotobuf-dev > version 3.6.1.3. > > > > Eugene > > > > -----Original Message----- > > From: Wei Mi <w...@google.com> > > Sent: Tuesday, May 25, 2021 8:07 PM > > To: Eugene Rozenfeld <eugene.rozenf...@microsoft.com> > > Cc: Andi Kleen <a...@linux.intel.com>; Hongtao Yu <h...@fb.com>; Xinliang > > David Li <davi...@google.com>; Jan Hubicka <hubi...@ucw.cz>; > > gcc@gcc.gnu.org; Wenlei He <wen...@fb.com> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > I checked the source of protobuf 3.0.0 and it didn't contain the > operator[] in RepeatedField. Need to install a newer version of protobuf. > > > > Thanks, > > Wei. > > > > On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld < > eugene.rozenf...@microsoft.com> wrote: > > > > > > Both are 3.0.0-9.1ubuntu1: > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list > > > protobuf-compiler Listing... Done protobuf-compiler/bionic,now > > > 3.0.0-9.1ubuntu1 amd64 [installed] > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list > > > libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now > > > 3.0.0-9.1ubuntu1 amd64 [installed] > > > > > > -----Original Message----- > > > From: Wei Mi <w...@google.com> > > > Sent: Tuesday, May 25, 2021 9:17 AM > > > To: Eugene Rozenfeld <eugene.rozenf...@microsoft.com> > > > Cc: Andi Kleen <a...@linux.intel.com>; Hongtao Yu <h...@fb.com>; > > > Xinliang David Li <davi...@google.com>; Jan Hubicka > > > <hubi...@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wen...@fb.com> > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > It looks like some version problem about protobuf-compiler and > libprotobuf-dev. Could you check what is the installed version on your end > for those two packages and see if they are consistent? > > > > > > On my platform, they are both 3.12.4. > > > > > > On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld < > eugene.rozenf...@microsoft.com> wrote: > > > > > > > > That eliminates the previous error but there is a new one: > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] > > > > Building CXX object > > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu > > > > ip > > > > pe > > > > r/perf_reader.cc.o > > > > FAILED: > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o > > > > /usr/bin/c++ -I../third_party/perf_data_converter/src > -I../third_party/perf_data_converter/src/quipper -I../ > -I../third_party/glog/src -I../third_party/abseil -I../util -I. > -Ithird_party/glog -std=gnu++1z -MD -MT > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o > -MF > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d > -o > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o > -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc > > > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In > member function 'bool > quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)': > > > > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: > error: no match for 'operator[]' (operand types are 'const > google::protobuf::RepeatedField<unsigned int>' and 'int') > > > > nrcpus = proto_uint32_metadata.data()[0]; > > > > > > > > -----Original Message----- > > > > From: Wei Mi <w...@google.com> > > > > Sent: Monday, May 24, 2021 8:54 PM > > > > To: Eugene Rozenfeld <eugene.rozenf...@microsoft.com> > > > > Cc: Andi Kleen <a...@linux.intel.com>; Hongtao Yu <h...@fb.com>; > > > > Xinliang David Li <davi...@google.com>; Jan Hubicka > > > > <hubi...@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wen...@fb.com> > > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > It isn't exposed on my platform either. Looks like a bug in > perf_data_converter (i.e., quipper). Could you try adding #include > <unordered_map> in > third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if > it fixes the problem? If it works, I will need to file a bug against > perf_data_converter. > > > > > > > > Thanks, > > > > Wei. > > > > > > > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld < > eugene.rozenf...@microsoft.com> wrote: > > > > > > > > > > That fixed the error I saw before but the build still fails. The > > > > > errors start with > > > > > > > > > > > > > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja > > > > > > > > > > [2/217] Building CXX object > > > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/ > > > > > qu > > > > > ip > > > > > pe > > > > > r/huge_page_deducer.cc.o > > > > > > > > > > FAILED: > > > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/ > > > > > qu > > > > > ip > > > > > pe > > > > > r/huge_page_deducer.cc.o > > > > > > > > > > /usr/bin/c++ -I../third_party/perf_data_converter/src > -I../third_party/perf_data_converter/src/quipper -I../ > -I../third_party/glog/src -I../third_party/abseil -I../util -I. > -Ithird_party/glog -std=gnu++1z -MD -MT > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o > -MF > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d > -o > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o > -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc > > > > > > > > > > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc: > > > > > 14 > > > > > 4:26: error: 'unordered_map' in namespace 'std' does not name a > > > > > template type > > > > > > > > > > using container = std::unordered_map<key_t, value_t>; > > > > > > > > > > ^~~~~~~~~~~~~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Wei Mi <w...@google.com> > > > > > Sent: Monday, May 24, 2021 8:12 PM > > > > > To: Eugene Rozenfeld <eugene.rozenf...@microsoft.com> > > > > > Cc: Andi Kleen <a...@linux.intel.com>; Hongtao Yu <h...@fb.com>; > > > > > Xinliang David Li <davi...@google.com>; Jan Hubicka > > > > > <hubi...@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wen...@fb.com> > > > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > > > > > > > > > > > > > Sorry, I added dependency for create_gcov but missed it for > dump_gcov. Fixed it at > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=826uc8So5JIW6bTbsxBMiEbLWexdiBiPjYrxexHOLmI%3D&reserved=0 > . > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Wei. > > > > > > > > > > > > > > > > > > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld < > eugene.rozenf...@microsoft.com> wrote: > > > > > > > > > > Thank you Wei. Looks like something is still missing. This time > perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 > with cmake 3.12.1): > > > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] > > > > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > > > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > > > > /usr/bin/c++ -I../ -I../third_party/glog/src > -I../third_party/abseil -I../third_party/perf_data_converter/src > -I../third_party/perf_data_converter/src/quipper -I../util -I. > -Ithird_party/glog -std=gnu++1z -MD -MT > CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF > CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o > CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc > > > > > In file included from > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > > > > from ../sample_reader.h:18, > > > > > from ../profile.h:15, > > > > > from ../profile.cc:5: > > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > > > > > > > In file included from ../profile.h:14:0, > > > > > from ../profile.cc:5: > > > > > ../base/macros.h:114:0: note: this is the location of the > > > > > previous definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) > > > > > \ > > > > > > > > > > In file included from > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > > > > from ../sample_reader.h:18, > > > > > from ../profile.h:15, > > > > > from ../profile.cc:5: > > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) > > > > > / > > > > > sizeof(*x)) > > > > > > > > > > In file included from ../profile.h:14:0, > > > > > from ../profile.cc:5: > > > > > ../base/macros.h:162:0: note: this is the location of the > > > > > previous definition #define arraysize(array) > > > > > (sizeof(ArraySizeHelper(array))) > > > > > > > > > > In file included from > ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0, > > > > > from ../sample_reader.h:18, > > > > > from ../profile.h:15, > > > > > from ../profile.cc:5: > > > > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: > > > > > fatal error: perf_data.pb.h: No such file or directory #include > "perf_data.pb.h" > > > > > ^~~~~~~~~~~~~~~~ > > > > > compilation terminated. > > > > > [6/241] Building CXX object > > > > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o > > > > > ninja: build stopped: subcommand failed. > > > > > > > > > > Thanks, > > > > > > > > > > Eugene > > > > > > > > > > From: Wei Mi <w...@google.com> > > > > > Sent: Saturday, May 22, 2021 9:37 AM > > > > > To: Eugene Rozenfeld <eugene.rozenf...@microsoft.com> > > > > > Cc: Andi Kleen <a...@linux.intel.com>; Hongtao Yu <h...@fb.com>; > > > > > Xinliang David Li <davi...@google.com>; Jan Hubicka > > > > > <hubi...@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wen...@fb.com> > > > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > > > It is a proto library build dependency issue which didn't expose > on my platform. I fix it at > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2Br9qK1fc2SQu%2FT9rHOCsJcYqzpkXnDu7SNfpLB9NCqQ%3D&reserved=0. > Please try again. > > > > > > > > > > Thanks, > > > > > Wei. > > > > > > > > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto: > eugene.rozenf...@microsoft.com> wrote: > > > > > I tried following the instructions in "2.2 Build autofdo tool for > gcc" in > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KpbQ1SBpwtPLdgJCpJQ6aW6iHj3mvisPN5kZVRiRQhM%3D&reserved=0 > got build failures: > > > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] > > > > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o > > > > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o > > > > > /usr/bin/c++ -I../ -I../third_party/glog/src > > > > > -I../third_party/abseil -I../third_party/perf_data_converter/src > -I../third_party/perf_data_converter/src/quipper -I../util -I. > -Ithird_party/glog -std=gnu++1z -MD -MT > CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF > CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o > CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file > included from > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > > > > from ../sample_reader.h:18, > > > > > from ../profile.h:15, > > > > > from ../profile.cc:5: > > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from > ../profile.h:14:0, > > > > > from ../profile.cc:5: > > > > > ../base/macros.h:114:0: note: this is the location of the > > > > > previous definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > > In file included from > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > > > > from ../sample_reader.h:18, > > > > > from ../profile.h:15, > > > > > from ../profile.cc:5: > > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > > > > sizeof(*x)) In file included from ../profile.h:14:0, > > > > > from ../profile.cc:5: > > > > > ../base/macros.h:162:0: note: this is the location of the > > > > > previous definition #define arraysize(array) > > > > > (sizeof(ArraySizeHelper(array))) In file included from > ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0, > > > > > from ../sample_reader.h:18, > > > > > from ../profile.h:15, > > > > > from ../profile.cc:5: > > > > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: > > > > > fatal error: perf_stat.pb.h: No such file or directory #include > "perf_stat.pb.h" > > > > > ^~~~~~~~~~~~~~~~ > > > > > compilation terminated. > > > > > > > > > > What is supposed to generate perf_stat.pb.h? > > > > > > > > > > Thanks, > > > > > > > > > > Eugene > > > > > > > > > > From: Wei Mi <mailto:w...@google.com> > > > > > Sent: Monday, May 10, 2021 4:47 PM > > > > > To: Andi Kleen <mailto:a...@linux.intel.com> > > > > > Cc: Hongtao Yu <mailto:h...@fb.com>; Xinliang David Li > > > > > <mailto:davi...@google.com>; Jan Hubicka > > > > > <mailto:hubi...@ucw.cz>; mailto:gcc@gcc.gnu.org; Eugene > > > > > Rozenfeld <mailto:eugene.rozenf...@microsoft.com>; Wenlei He > > > > > <mailto:wen...@fb.com> > > > > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258135446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0qVyYYmYDZfHt3cdxg1LjUYvonck6G6PLAOFeYCNnTc%3D&reserved=0 > has been updated. Now create_gcov/dump_gcov are added back and can be built > separately. > > > > > > > > > > Please look at "2.2 Build autofdo tool for gcc" in > > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% > > > > > 2F > > > > > gi > > > > > th > > > > > ub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Roz > > > > > en > > > > > fe > > > > > ld > > > > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86 > > > > > f1 > > > > > 41 > > > > > af > > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbG > > > > > Zs > > > > > b3 > > > > > d8 > > > > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% > > > > > 3D > > > > > %7 > > > > > C1 > > > > > 000&sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&a > > > > > mp > > > > > ;r > > > > > es > > > > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen > > > > > <mailto:a...@linux.intel.com> wrote: > > > > > > > > > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote: > > > > > > > Andi, thanks for pointing out the perf script issues. Can > you please > > > > > > > elaborate a bit on the exact issue you have seen? We've > been using > > > > > > > specific output of perf script such as mmap, LBR and > callstack events > > > > > > > filtered by process id. It works fine so far but may > certainly hit issues > > > > > > > in the future with extended uses. > > > > > > > > > > > > Okay I took a look at the latest autofdo now. It seems to be > > > > > > basically a LLVM project now that depends on LLVM to even > > > > > > build with all kinds of dependency hell on some old LLVM version > and other packages. > > > > > > > > > > > > I guess gcc will really need a replacement that doesn't pull > > > > > > in all of LLVM if it wants to continue supporting autofdo. > > > > > > > > > > > > I'm myself unable to build now. > > > > > > > > > > > > I'm using the old version I had a git fork of and that was > > > > > > before all of this. I added a patch to make it work with the > > > > > > latest perf by ignoring increased perf_attr and unknown perf > events. > > > > > > > > > > > > Honza please use > > > > > > > > > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 > > > > > > F% > > > > > > 2F > > > > > > gi > > > > > > thub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozen > > > > > > fe > > > > > > ld > > > > > > %4 > > > > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86 > > > > > > f1 > > > > > > 41 > > > > > > af > > > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFp > > > > > > bG > > > > > > Zs > > > > > > b3 > > > > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 > > > > > > Mn > > > > > > 0% > > > > > > 3D > > > > > > %7C1000&sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8 > > > > > > g% > > > > > > 3D > > > > > > &a > > > > > > mp;reserved=0 -b perf-future > > > > > > > > > > > > for testing. > > > > > > > > > > > > -Andi > > > > > > >
