> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L > <nathaniel.l.desim...@intel.com> wrote: > > Hi Liming, > > I chatted with Aaron on the phone today. The VfrCompiler race condition was > discovered using "make -j <n>" (where n > 1). I have filed the bug in > Bugzilla: >
I think there are a few issues like that in BaseTools. We have a parallel makefile and we had to turn off parallelism when calling the BaseTools. In general the edk2 build system fights against "make -j <n>" since the Python build command is trying to do the threading and invoking make in parallel. Given we ported EDK to parallel build, the Python parallel edk2 build is a lot slower than the EDK make parallel build, even when adjusting for the extra work the edk2 does. I even noticed a 1,000,000 calls to regex in Python during the ... phase of the build. I think the correct short term fix may be to put .NOTPARALLEL: in the BaseTools GNUmakefile given it is constructed in a way that it does not support parallelism. Thanks, Andrew Fish > https://bugzilla.tianocore.org/show_bug.cgi?id=786 > > For the ARCH environment variable, we brainstormed two possible new names > HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable request in > Bugzilla as well? > > Thanks, > > Nate > > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of Gao, > Liming > Sent: Monday, November 13, 2017 5:46 PM > To: Aaron Durbin <adur...@google.com> > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > I add my comments. > >> -----Original Message----- >> From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of >> Aaron Durbin >> Sent: Tuesday, November 14, 2017 1:38 AM >> To: Gao, Liming <liming....@intel.com> >> Cc: edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming....@intel.com> wrote: >>> Yes. BaseTools needs some environments. Before build BaseTools, we need to >>> call edkset.sh to setup them. >>> >>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >> built first. I agree that this way takes some time to compile >> BaseTools source. I have one proposal to compile C source tools with >> multiple thread. I can share my draft patch. >> >> If the depedencies are correctly met in the makefiles then why are >> they built serially? It sounds like you understand the dependencies so >> why aren't those explicitly noted so one can build in parallel? >> > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you > submit this issue in bugzillar (https://bugzilla.tianocore.org/)? > Besides, do you use make -j option to enable Parallel Execution? I want to > know how to verify it. > >>> 2. BaseTools C source are compiled to the executable files. They run >>> in host machine. Here ARCH is for the executable files those can >> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH >> based on 'uname -m' command. >> >> ARCH != host is my point. Why are those being conflated? Or are you >> saying edk2 tools define ARCH to be what the host is? >> > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too > generic. In your environment, ARCH may be defined for the different purpose. > >>> 3. There are more GCC compiler distribution. We try to use the >>> generic options in GCC tool chain. If you find the option doesn't >>> work >> on your GCC compiler, please report it. And, we also notice >> tools_def.txt is too big to be hard to be maintain. We have one proposal to >> simplify it. I will send this RFC to edk2 community. >>> >>>> -----Original Message----- >>>> From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf >>>> Of Aaron Durbin >>>> Sent: Saturday, November 11, 2017 6:27 AM >>>> To: edk2-devel@lists.01.org >>>> Subject: [edk2] EDK2 build issues >>>> >>>> Hello, >>>> >>>> Here are some observations I've encountered trying to utilize edk2 >>>> for certain builds. Part of the problem seems to be with implicit >>>> assumptions in how edk2 is used. I'm trying to build things using >>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>> isn't a workspace that exists on one persons computer for the >>>> lifetime of development. >>>> >>>> 1. BaseTools can't build in parallel. The tests are racey which >>>> result in test failures. Because of this one has to build these in >>>> serial instead of in parallel. >>>> >>>> =========================================================== >>>> =========== >>>> FAIL: testHelp (TianoCompress.Tests) >>>> -------------------------------------------------------------------- >>>> -- >>>> Traceback (most recent call last): >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 34, in testHelp >>>> self.assertTrue(result == 0) >>>> AssertionError: False is not true >>>> >>>> =========================================================== >>>> =========== >>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>> -------------------------------------------------------------------- >>>> -- >>>> Traceback (most recent call last): >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 65, in testRandomDataCycles >>>> self.compressionTestCycle(data) >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 44, in compressionTestCycle >>>> self.assertTrue(result == 0) >>>> AssertionError: False is not true >>>> >>>> In addition, it seems compilation even breaks trying to build a parser: >>>> >>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>> ^ >>>> VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>> field named 'VfrLexer' >>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>> ^ >>>> >>>> This just slows down builds needing to things serially. >>>> >>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>> the *host* you are trying to build tools on -- not the target. >>>> Trying to incorporate edk2 builds into a portage environment >>>> effectively breaks because of this as ARCH refers to target >>>> architecture -- not host builder's ARCH. >>>> >>>> 3. This more of an observation, but the tools definition seems to >>>> make quite the leap on how consistent compilers are of a certain version. >>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>> edk2 implicitly assumes are set based on some distribution's default flags? >>>> And in order to extend toolchain support one needs to create a >>>> series of entries associated with a certain family. Barrier to entry >>>> is pretty high in teasing out where to put things in the ~8000 line >>>> tools_def.template. >>>> >>>> Thanks for the consideration. >>>> >>>> -Aaron >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
