Both are 3.0.0-9.1ubuntu1: eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list protobuf-compiler Listing... Done protobuf-compiler/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed] eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
-----Original Message----- From: Wei Mi <[email protected]> Sent: Tuesday, May 25, 2021 9:17 AM To: Eugene Rozenfeld <[email protected]> Cc: Andi Kleen <[email protected]>; Hongtao Yu <[email protected]>; Xinliang David Li <[email protected]>; Jan Hubicka <[email protected]>; [email protected]; Wenlei He <[email protected]> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent? On my platform, they are both 3.12.4. On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <[email protected]> wrote: > > That eliminates the previous error but there is a new one: > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] Building > CXX object > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe > r/perf_reader.cc.o > FAILED: > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o > /usr/bin/c++ -I../third_party/perf_data_converter/src > -I../third_party/perf_data_converter/src/quipper -I../ > -I../third_party/glog/src -I../third_party/abseil -I../util -I. > -Ithird_party/glog -std=gnu++1z -MD -MT > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o > -MF > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d > -o > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o > -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member > function 'bool > quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)': > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: > no match for 'operator[]' (operand types are 'const > google::protobuf::RepeatedField<unsigned int>' and 'int') > nrcpus = proto_uint32_metadata.data()[0]; > > -----Original Message----- > From: Wei Mi <[email protected]> > Sent: Monday, May 24, 2021 8:54 PM > To: Eugene Rozenfeld <[email protected]> > Cc: Andi Kleen <[email protected]>; Hongtao Yu <[email protected]>; Xinliang > David Li <[email protected]>; Jan Hubicka <[email protected]>; > [email protected]; Wenlei He <[email protected]> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > It isn't exposed on my platform either. Looks like a bug in > perf_data_converter (i.e., quipper). Could you try adding #include > <unordered_map> in > third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if > it fixes the problem? If it works, I will need to file a bug against > perf_data_converter. > > Thanks, > Wei. > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld > <[email protected]> wrote: > > > > That fixed the error I saw before but the build still fails. The > > errors start with > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja > > > > [2/217] Building CXX object > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip > > pe > > r/huge_page_deducer.cc.o > > > > FAILED: > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip > > pe > > r/huge_page_deducer.cc.o > > > > /usr/bin/c++ -I../third_party/perf_data_converter/src > > -I../third_party/perf_data_converter/src/quipper -I../ > > -I../third_party/glog/src -I../third_party/abseil -I../util -I. > > -Ithird_party/glog -std=gnu++1z -MD -MT > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o > > -MF > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d > > -o > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o > > -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc > > > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc: > > 14 > > 4:26: error: 'unordered_map' in namespace 'std' does not name a > > template type > > > > using container = std::unordered_map<key_t, value_t>; > > > > ^~~~~~~~~~~~~ > > > > > > > > > > > > > > > > From: Wei Mi <[email protected]> > > Sent: Monday, May 24, 2021 8:12 PM > > To: Eugene Rozenfeld <[email protected]> > > Cc: Andi Kleen <[email protected]>; Hongtao Yu <[email protected]>; > > Xinliang David Li <[email protected]>; Jan Hubicka > > <[email protected]>; [email protected]; Wenlei He <[email protected]> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. > > Fixed it at > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315397315%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9G9uGhG%2BmK35bdV4QL9BOTVg6bFFQrVP46YdLOi3Ak4%3D&reserved=0. > > > > > > > > Thanks, > > > > Wei. > > > > > > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld > > <[email protected]> wrote: > > > > Thank you Wei. Looks like something is still missing. This time > > perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 > > with cmake 3.12.1): > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil > > -I../third_party/perf_data_converter/src > > -I../third_party/perf_data_converter/src/quipper -I../util -I. > > -Ithird_party/glog -std=gnu++1z -MD -MT > > CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF > > CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o > > CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc > > In file included from > > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > In file included from ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:114:0: note: this is the location of the previous > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > In file included from > > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > sizeof(*x)) > > > > In file included from ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:162:0: note: this is the location of the previous > > definition #define arraysize(array) > > (sizeof(ArraySizeHelper(array))) > > > > In file included from > > ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: > > fatal error: perf_data.pb.h: No such file or directory #include > > "perf_data.pb.h" > > ^~~~~~~~~~~~~~~~ > > compilation terminated. > > [6/241] Building CXX object > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o > > ninja: build stopped: subcommand failed. > > > > Thanks, > > > > Eugene > > > > From: Wei Mi <[email protected]> > > Sent: Saturday, May 22, 2021 9:37 AM > > To: Eugene Rozenfeld <[email protected]> > > Cc: Andi Kleen <[email protected]>; Hongtao Yu <[email protected]>; > > Xinliang David Li <[email protected]>; Jan Hubicka > > <[email protected]>; [email protected]; Wenlei He <[email protected]> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > It is a proto library build dependency issue which didn't expose on my > > platform. I fix it at > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SL5mLhelrkeB26Mg7tNaml0qjZ2OklYPwQR9btVhsqk%3D&reserved=0. > > Please try again. > > > > Thanks, > > Wei. > > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld > > <mailto:[email protected]> wrote: > > I tried following the instructions in "2.2 Build autofdo tool for gcc" in > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PYVm%2BHsRk%2Bfw7%2BaNgYNWn8zMnBsqcqqVSpyXQktd7sA%3D&reserved=0 > > got build failures: > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o > > /usr/bin/c++ -I../ -I../third_party/glog/src > > -I../third_party/abseil -I../third_party/perf_data_converter/src > > -I../third_party/perf_data_converter/src/quipper -I../util -I. > > -Ithird_party/glog -std=gnu++1z -MD -MT > > CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF > > CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o > > CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file > > included from > > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from > > ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:114:0: note: this is the location of the previous > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file > > included from > > ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > sizeof(*x)) In file included from ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:162:0: note: this is the location of the previous > > definition #define arraysize(array) (sizeof(ArraySizeHelper(array))) > > In file included from > > ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: > > fatal error: perf_stat.pb.h: No such file or directory #include > > "perf_stat.pb.h" > > ^~~~~~~~~~~~~~~~ > > compilation terminated. > > > > What is supposed to generate perf_stat.pb.h? > > > > Thanks, > > > > Eugene > > > > From: Wei Mi <mailto:[email protected]> > > Sent: Monday, May 10, 2021 4:47 PM > > To: Andi Kleen <mailto:[email protected]> > > Cc: Hongtao Yu <mailto:[email protected]>; Xinliang David Li > > <mailto:[email protected]>; Jan Hubicka <mailto:[email protected]>; > > mailto:[email protected]; Eugene Rozenfeld > > <mailto:[email protected]>; Wenlei He > > <mailto:[email protected]> > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uwQwWdjE%2FL%2F2r0P%2Bdn7r5ulE8kRmPPw8A2XOQuZnUhM%3D&reserved=0 > > has been updated. Now create_gcov/dump_gcov are added back and can be > > built separately. > > > > Please look at "2.2 Build autofdo tool for gcc" in > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi > > th > > ub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfe > > ld > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141 > > af > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3 > > d8 > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7 > > C1 > > 000&sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&r > > es > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen > > <mailto:[email protected]> wrote: > > > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote: > > > > Andi, thanks for pointing out the perf script issues. Can you please > > > > elaborate a bit on the exact issue you have seen? We've been using > > > > specific output of perf script such as mmap, LBR and callstack events > > > > filtered by process id. It works fine so far but may certainly hit > > > > issues > > > > in the future with extended uses. > > > > > > Okay I took a look at the latest autofdo now. It seems to be > > > basically a LLVM project now that depends on LLVM to even build > > > with all kinds of dependency hell on some old LLVM version and other > > > packages. > > > > > > I guess gcc will really need a replacement that doesn't pull in > > > all of LLVM if it wants to continue supporting autofdo. > > > > > > I'm myself unable to build now. > > > > > > I'm using the old version I had a git fork of and that was before > > > all of this. I added a patch to make it work with the latest perf > > > by ignoring increased perf_attr and unknown perf events. > > > > > > Honza please use > > > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > gi > > > thub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld > > > %4 > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141 > > > af > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs > > > b3 > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% > > > 3D > > > %7C1000&sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D > > > &a > > > mp;reserved=0 -b perf-future > > > > > > for testing. > > > > > > -Andi > > >
