Hi. I have submitted a pull request to edk2-basetools repository: https://github.com/tianocore/edk2-basetools/pull/88 This is the feature request for it: https://github.com/tianocore/edk2-basetools/issues/87 I'm also attaching the patch here: (0001-BaseTools-Generate-compile-information-in-build-repo.patch)
On a side note, seems like tip of edk2-basetools is broken due this commit that makes direct import of Common package: https://github.com/tianocore/edk2-basetools/commit/8e6018d3ea4c1aae7185f589d129cea14a5d89fd edk2-basetools\edk2basetools\GenFds\SubTypeGuidSection.py: import Common.LongFilePathOs as os -----Original Message----- From: Ni, Ray <ray...@intel.com> Sent: Wednesday, February 15, 2023 6:51 PM To: Kinney, Michael D <michael.d.kin...@intel.com>; devel@edk2.groups.io; Chen, Christine <yuwei.c...@intel.com>; Palomino Sosa, Guillermo A <guillermo.a.palomino.s...@intel.com> Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming <gaolim...@byosoft.com.cn> Subject: RE: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile information in build report Mike, Thanks for the explanation of status and plan. > -----Original Message----- > From: Kinney, Michael D <michael.d.kin...@intel.com> > Sent: Thursday, February 16, 2023 7:13 AM > To: Ni, Ray <ray...@intel.com>; devel@edk2.groups.io; Chen, Christine > <yuwei.c...@intel.com>; Palomino Sosa, Guillermo A > <guillermo.a.palomino.s...@intel.com> > Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming > <gaolim...@byosoft.com.cn>; Kinney, Michael D > <michael.d.kin...@intel.com> > Subject: RE: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile > information in build report > > Hi Ray, > > Right now we want the commit in both places. > > The priority is edk2-basetools first. It has more CI checks than edk2 > repo for tools and packages up as a pip module. > > As soon as edk2-basetools change is merged, the edk2 repo change can > be submitted and merged because the reviews have already been > completed by the BaseTools maintainers. > > We do not like commit in 2 places. As soon as possible, we would like > to see the python sources removed from edk2/BaseTools and all devs/CI > only use edk2- basetools for python based tools. > > Mike > > > -----Original Message----- > > From: Ni, Ray <ray...@intel.com> > > Sent: Wednesday, February 15, 2023 12:17 AM > > To: devel@edk2.groups.io; Chen, Christine <yuwei.c...@intel.com>; > Kinney, Michael D <michael.d.kin...@intel.com>; Palomino Sosa, > > Guillermo A <guillermo.a.palomino.s...@intel.com> > > Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming > <gaolim...@byosoft.com.cn> > > Subject: RE: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile > information in build report > > > > Christine, > > If BaseTools related changes is implemented in edk2-basetools repo, > > does that mean if I only checkout edk2 repo, I am using an older > > version of > BaseTools? > > > > Thanks, > > Ray > > > > > -----Original Message----- > > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of > Yuwei > > > Chen > > > Sent: Wednesday, February 15, 2023 1:38 PM > > > To: Kinney, Michael D <michael.d.kin...@intel.com>; > devel@edk2.groups.io; > > > Palomino Sosa, Guillermo A <guillermo.a.palomino.s...@intel.com> > > > Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming > > > <gaolim...@byosoft.com.cn> > > > Subject: Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate > > > compile information in build report > > > > > > Hi Mike, thanks for reminder. > > > > > > Hi Willy, currently, BaseTools related changes will be implemented > > > on the edk2-basetools repo. Please send the patch based on the > > > edk2-basetools repo~ > > > > > > Thanks, > > > Christine > > > > > > > -----Original Message----- > > > > From: Kinney, Michael D <michael.d.kin...@intel.com> > > > > Sent: Wednesday, February 15, 2023 11:43 AM > > > > To: devel@edk2.groups.io; Chen, Christine > > > > <yuwei.c...@intel.com>; Palomino Sosa, Guillermo A > > > > <guillermo.a.palomino.s...@intel.com> > > > > Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming > > > > <gaolim...@byosoft.com.cn>; Kinney, Michael D > > > > <michael.d.kin...@intel.com> > > > > Subject: RE: [edk2-devel][PATCH V1 1/1] BaseTools: Generate > > > > compile information in build report > > > > > > > > Has this been reviewed for edk2-basetools repo? > > > > > > > > Mike > > > > > > > > > -----Original Message----- > > > > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of > > > Yuwei > > > > > Chen > > > > > Sent: Tuesday, February 14, 2023 6:44 PM > > > > > To: Palomino Sosa, Guillermo A > <guillermo.a.palomino.s...@intel.com>; > > > > > devel@edk2.groups.io > > > > > Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming > > > > > <gaolim...@byosoft.com.cn> > > > > > Subject: Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate > compile > > > > > information in build report > > > > > > > > > > Reviewed-by: Yuwei Chen <yuwei.c...@intel.com> > > > > > > > > > > > -----Original Message----- > > > > > > From: Palomino Sosa, Guillermo A > > > > > > <guillermo.a.palomino.s...@intel.com> > > > > > > Sent: Tuesday, February 7, 2023 11:07 AM > > > > > > To: devel@edk2.groups.io > > > > > > Cc: Feng, Bob C <bob.c.f...@intel.com>; Gao, Liming > > > > > > <gaolim...@byosoft.com.cn>; Chen, Christine > <yuwei.c...@intel.com> > > > > > > Subject: [edk2-devel][PATCH V1 1/1] BaseTools: Generate > > > > > > compile information in build report > > > > > > > > > > > > REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2850 > > > > > > > > > > > > Add "-Y REPORT_INFO" option to build command to generate > compile > > > > > > information as part of BuildReport. > > > > > > This option generates files to be used by external tools as > > > > > > IDE's to enhance functionality. > > > > > > Files are created inside build folder: > > > > > > <Build>/<BuildTarget>/<ToolChain>/CompileInfo > > > > > > > > > > > > Files created: > > > > > > * compile_commands.json - Compilation Database. To be used > > > > > > by > IDE's > > > > > > to enable advance features > > > > > > * cscope.files - List of files used in compilation. Used by > > > > > > Cscope to > parse > > > > > > C code and provide browse functionality. > > > > > > * module_report.json - Module data form buildReport in Json > format. > > > > > > > > > > > > Signed-off-by: Guillermo Antonio Palomino Sosa > > > > > > <guillermo.a.palomino.s...@intel.com> > > > > > > --- > > > > > > BaseTools/Source/Python/build/BuildReport.py | 139 > > > > > > +++++++++++++++++++- > > > > > > BaseTools/Source/Python/build/buildoptions.py | 4 +- > > > > > > 2 files changed, 140 insertions(+), 3 deletions(-) > > > > > > > > > > > > diff --git a/BaseTools/Source/Python/build/BuildReport.py > > > > > > b/BaseTools/Source/Python/build/BuildReport.py > > > > > > index 468772930c..33b43d471f 100644 > > > > > > --- a/BaseTools/Source/Python/build/BuildReport.py > > > > > > +++ b/BaseTools/Source/Python/build/BuildReport.py > > > > > > @@ -10,6 +10,8 @@ > > > > > > > > > > > > ## Import Modules > > > > > > # > > > > > > +import json > > > > > > +from pathlib import Path > > > > > > import Common.LongFilePathOs as os import re import > > > > > > platform @@ -41,6 +43,7 @@ from Common.DataType import * > > > > > > import > > > > collections > > > > > > from Common.Expression import * from GenFds.AprioriSection > import > > > > > > DXE_APRIORI_GUID, PEI_APRIORI_GUID > > > > > > +from AutoGen.IncludesAutoGen import IncludesAutoGen > > > > > > > > > > > > ## Pattern to extract contents in EDK DXS files > > > > > > gDxsDependencyPattern = > > > > > > re.compile(r"DEPENDENCY_START(.+)DEPENDENCY_END", > re.DOTALL) > > > > @@ - > > > > > > 2298,6 +2301,10 @@ class BuildReport(object): > > > > > > def GenerateReport(self, BuildDuration, AutoGenTime, > MakeTime, > > > > > > GenFdsTime): > > > > > > if self.ReportFile: > > > > > > try: > > > > > > + > > > > > > + if "COMPILE_INFO" in self.ReportType: > > > > > > + self.GenerateCompileInfo() > > > > > > + > > > > > > File = [] > > > > > > for (Wa, MaList) in self.ReportList: > > > > > > PlatformReport(Wa, MaList, > > > > > > self.ReportType).GenerateReport(File, BuildDuration, > AutoGenTime, > > > > > > MakeTime, GenFdsTime, self.ReportType) @@ -2310,7 +2317,137 > @@ > > > > class > > > > > > BuildReport(object): > > > > > > EdkLogger.error("BuildReport", CODE_ERROR, > > > > > > "Unknown fatal error when generating build report", > > > > > > ExtraData=self.ReportFile, > > > > > > RaiseError=False) > > > > > > EdkLogger.quiet("(Python %s on %s\n%s)" % > > > > > > (platform.python_version(), sys.platform, > > > > > > traceback.format_exc())) > > > > > > > > > > > > + > > > > > > + ## > > > > > > + # Generates compile data files to be used by external tools. > > > > > > + # Compile information will be generated in > > > > > > <Build>/<BuildTarget>/<ToolChain>/CompileInfo > > > > > > + # Files generated: compile_commands.json, cscope.files, > > > > > > modules_report.json > > > > > > + # > > > > > > + # @param self The object pointer > > > > > > + # > > > > > > + def GenerateCompileInfo(self): > > > > > > + try: > > > > > > + # Lists for the output elements > > > > > > + compile_commands = [] > > > > > > + used_files = set() > > > > > > + module_report = [] > > > > > > + > > > > > > + for (Wa, MaList) in self.ReportList: > > > > > > + # Obtain list of all processed Workspace files > > > > > > + for file_path in > > > > > > + Wa._GetMetaFiles(Wa.BuildTarget, > > > > Wa.ToolChain): > > > > > > + used_files.add(file_path) > > > > > > + > > > > > > + for autoGen in Wa.AutoGenObjectList: > > > > > > + > > > > > > + # Loop through all modules > > > > > > + for module in > > > > > > + (autoGen.LibraryAutoGenList + > > > > > > autoGen.ModuleAutoGenList): > > > > > > + > > > > > > + > > > > > > + used_files.add(module.MetaFile.Path) > > > > > > + > > > > > > + # Main elements of module report > > > > > > + module_report_data = {} > > > > > > + module_report_data["Name"] = module.Name > > > > > > + module_report_data["Arch"] = module.Arch > > > > > > + module_report_data["Path"] = > > > > > > module.MetaFile.Path > > > > > > + module_report_data["Guid"] = module.Guid > > > > > > + module_report_data["BuildType"] = > module.BuildType > > > > > > + module_report_data["IsLibrary"] = > > > > > > module.IsLibrary > > > > > > + module_report_data["SourceDir"] = > > > > > > module.SourceDir > > > > > > + module_report_data["Files"] = [] > > > > > > + > > > > > > + # Files used by module > > > > > > + for data_file in module.SourceFileList: > > > > > > + > > > > > > module_report_data["Files"].append({"Name": > > > > > > + data_file.Name, "Path": data_file.Path}) > > > > > > + > > > > > > + # Libraries used by module > > > > > > + module_report_data["Libraries"] = [] > > > > > > + for data_library in > > > > > > module.LibraryAutoGenList: > > > > > > + > > > > > > + module_report_data["Libraries"].append({"Path": > > > > > > + data_library.MetaFile.Path}) > > > > > > + > > > > > > + # Packages used by module > > > > > > + module_report_data["Packages"] = [] > > > > > > + for data_package in module.PackageList: > > > > > > + > > > > > > module_report_data["Packages"].append({"Path": > > > > > > data_package.MetaFile.Path, "Includes": []}) > > > > > > + # Includes path used in package > > > > > > + for data_package_include in > > > > > > data_package.Includes: > > > > > > + > > > > > > + module_report_data["Packages"][- > > > > > > 1]["Includes"].append(data_package_inc > > > > > > + lude.Path) > > > > > > + > > > > > > + # PPI's in module > > > > > > + module_report_data["PPI"] = [] > > > > > > + for data_ppi in module.PpiList.keys(): > > > > > > + > > > > > > module_report_data["PPI"].append({"Name": > > > > > > + data_ppi, "Guid": module.PpiList[data_ppi]}) > > > > > > + > > > > > > + # Protocol's in module > > > > > > + module_report_data["Protocol"] = [] > > > > > > + for data_protocol in > > > > > > module.ProtocolList.keys(): > > > > > > + > > > > > > + module_report_data["Protocol"].append({"Name": > data_protocol, > > > > "Guid": > > > > > > + module.ProtocolList[data_protocol]}) > > > > > > + > > > > > > + # PCD's in module > > > > > > + module_report_data["Pcd"] = [] > > > > > > + for data_pcd in module.LibraryPcdList: > > > > > > + > > > > > > module_report_data["Pcd"].append({"Space": > > > > > > data_pcd.TokenSpaceGuidCName, > > > > > > + > > > > > > "Name": data_pcd.TokenCName, > > > > > > + > > > > > > "Value": data_pcd.TokenValue, > > > > > > + > > > > > > "Guid": > > > > data_pcd.TokenSpaceGuidValue, > > > > > > + > > > > > > "DatumType": > data_pcd.DatumType, > > > > > > + > > > > > > "Type": data_pcd.Type, > > > > > > + > > > > > > "DefaultValue": > > > > data_pcd.DefaultValue}) > > > > > > + # Add module to report > > > > > > + > > > > > > + module_report.append(module_report_data) > > > > > > + > > > > > > + # Include file dependencies to used files > > > > > > + includes_autogen = > > > > > > + IncludesAutoGen(module.MakeFileDir, > > > > > > module) > > > > > > + for dep in includes_autogen.DepsCollection: > > > > > > + used_files.add(dep) > > > > > > + > > > > > > + inc_flag = "-I" # Default include flag > > > > > > + if module.BuildRuleFamily == > > > > > > TAB_COMPILER_MSFT: > > > > > > + inc_flag = "/I" > > > > > > + > > > > > > + for source in module.SourceFileList: > > > > > > + used_files.add(source.Path) > > > > > > + compile_command = {} > > > > > > + if source.Ext in [".c", ".cc", ".cpp"]: > > > > > > + # > > > > > > + # Generate compile command for > > > > > > each c file > > > > > > + # > > > > > > + compile_command["file"] = > > > > > > source.Path > > > > > > + compile_command["directory"] = > > > > > > source.Dir > > > > > > + build_command = > > > > > > module.BuildRules[source.Ext].CommandList[0] > > > > > > + build_command_variables = > > > > > > + re.findall(r"\$\((.*?)\)", > > > > > > build_command) > > > > > > + for var in build_command_variables: > > > > > > + var_tokens = var.split("_") > > > > > > + var_main = var_tokens[0] > > > > > > + if len(var_tokens) == 1: > > > > > > + var_value = > > > > module.BuildOption[var_main]["PATH"] > > > > > > + else: > > > > > > + var_value = > > > > > > module.BuildOption[var_main][var_tokens[1]] > > > > > > + build_command = > > > > > > + build_command.replace(f"$({var})", > > > > > > var_value) > > > > > > + include_files = f" > > > > {inc_flag}".join(module.IncludePathList) > > > > > > + build_command = > > > > > > + build_command.replace("${src}", > > > > > > include_files) > > > > > > + build_command = > > > > > > + build_command.replace("${dst}", module.OutputDir) > > > > > > + > > > > > > + # Remove un defined macros > > > > > > + compile_command["command"] > > > > > > + = re.sub(r"\$\(.*?\)", "", > > > > > > build_command) > > > > > > + > > > > > > + compile_commands.append(compile_command) > > > > > > + > > > > > > + # Create output folder if doesn't exist > > > > > > + compile_info_folder = > > > > Path(Wa.BuildDir).joinpath("CompileInfo") > > > > > > + compile_info_folder.mkdir(exist_ok=True) > > > > > > + > > > > > > + # Sort and save files > > > > > > + compile_commands.sort(key=lambda x: > > > > > > + x["file"]) > > > > > > + > > > > > > + > > > > > > > > > > > > > > SaveFileOnChange(compile_info_folder.joinpath(f"compile_commands.json > > > " > > > > > > + ),json.dumps(compile_commands, indent=2), False) > > > > > > + > > > > > > + > > > > > > + SaveFileOnChange(compile_info_folder.joinpath(f"cscope.fil > > > > > > + es"), "\n".join(sorted(used_files)), False) > > > > > > + > > > > > > + module_report.sort(key=lambda x: x["Path"]) > > > > > > + > > > > > > + > > > SaveFileOnChange(compile_info_folder.joinpath(f"module_report.json > > > > > > + "), json.dumps(module_report, indent=2), False) > > > > > > + > > > > > > + except: > > > > > > + EdkLogger.error("BuildReport", CODE_ERROR, > > > > > > + "Unknown fatal error > > > > > > when generating build report compile information", > > > > > > ExtraData=self.ReportFile, RaiseError=False) > > > > > > + EdkLogger.quiet("(Python %s on %s\n%s)" % > > > > > > + (platform.python_version(), sys.platform, > traceback.format_exc())) > > > > > > + > > > > > > # This acts like the main() function for the script, unless > > > > > > it is 'import'ed into another script. > > > > > > if __name__ == '__main__': > > > > > > pass > > > > > > - > > > > > > diff --git a/BaseTools/Source/Python/build/buildoptions.py > > > > > > b/BaseTools/Source/Python/build/buildoptions.py > > > > > > index 8334604b46..5ec561f7ec 100644 > > > > > > --- a/BaseTools/Source/Python/build/buildoptions.py > > > > > > +++ b/BaseTools/Source/Python/build/buildoptions.py > > > > > > @@ -84,8 +84,8 @@ class MyOptionParser(): > > > > > > Parser.add_option("-D", "--define", > > > > > > action="append", type="string", dest="Macros", help="Macro: > > > > > > \"Name [= Value]\".") > > > > > > > > > > > > Parser.add_option("-y", "--report-file", > > > > > > action="store", dest="ReportFile", help="Create/overwrite > > > > > > the report to the specified > > > > > > filename.") > > > > > > - Parser.add_option("-Y", "--report-type", action="append", > > > > > > type="choice", choices=['PCD', 'LIBRARY', 'FLASH', 'DEPEX', > > > > > > 'BUILD_FLAGS', 'FIXED_ADDRESS', 'HASH', 'EXECUTION_ORDER'], > > > > > > dest="ReportType", default=[], > > > > > > - help="Flags that control the type of build report to > > > > > > generate. > > > > Must be > > > > > > one of: [PCD, LIBRARY, FLASH, DEPEX, BUILD_FLAGS, > FIXED_ADDRESS, > > > > > > HASH, EXECUTION_ORDER]. "\ > > > > > > + Parser.add_option("-Y", "--report-type", > > > > > > + action="append", > > > > > > type="choice", choices=['PCD', 'LIBRARY', 'FLASH', 'DEPEX', > > > > > > 'BUILD_FLAGS', 'FIXED_ADDRESS', 'HASH', 'EXECUTION_ORDER', > > > > > > 'COMPILE_INFO'], dest="ReportType", default=[], > > > > > > + help="Flags that control the type of build > > > > > > + report to generate. Must be one of: [PCD, LIBRARY, FLASH, > > > > > > + DEPEX, BUILD_FLAGS, FIXED_ADDRESS, HASH, EXECUTION_ORDER, > > > > COMPILE_INFO]. > > > > > > + "\ > > > > > > "To specify more than one flag, repeat > > > > > > this option on the command line and the default flag set is > > > > > > [PCD, LIBRARY, FLASH, DEPEX, HASH, BUILD_FLAGS, FIXED_ADDRESS]") > > > > > > Parser.add_option("-F", "--flag", action="store", > > > > > > type="string", dest="Flag", > > > > > > help="Specify the specific option to parse EDK > > > > > > UNI file. Must > be > > > > one of: > > > > > > [-c, -s]. -c is for EDK framework UNI file, and -s is for > > > > > > EDK UEFI UNI file. "\ > > > > > > -- > > > > > > 2.28.0.windows.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#100516): https://edk2.groups.io/g/devel/message/100516 Mute This Topic: https://groups.io/mt/96800354/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
0001-BaseTools-Generate-compile-information-in-build-repo.patch
Description: 0001-BaseTools-Generate-compile-information-in-build-repo.patch
