Re: [easybuild] GCCcore build in HMNS
qu...@lists.ugent.be [mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Vanzo, Davide Sent: Wednesday, March 8, 2017 2:09 PM To: easybuild@lists.ugent.be Subject: Re: [easybuild] GCCcore build in HMNS Shahzeb, Looks good to me. It is expected that GCCcore depends on dummy toolchain built software since that is the only toolchain below GCCcore. In other words, you need whatever has been built with the system compiler and binutils in order to compile GCCcore. Once you have GCCcore built, at this point you will use it to compile again M4/Bison/flex/zlib/binutils within the GCCcore toolchain so that whatever is built with GCC is completely independent from the underlying OS software stack. -- Davide Vanzo, PhD Application Developer Adjunct Assistant Professor of Chemical and Biomolecular Engineering Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 www.accre.vanderbilt.edu<http://www.accre.vanderbilt.edu> On Mar 8 2017, at 12:58 pm, Siddiqui, Shahzeb mailto:shahzeb.siddi...@pfizer.com>> wrote: Hello, I want to find out if the build for GCCcore 5.4.0 is suppose to look like this under HMNS. I have copied all the files from easybuild repo into my local directory because I have disabled robot paths. One thing I don’t understand why there is M4 and binutils built with GCCcore toolchain while it also is a dependency in GCCcore. When I tried to build GCCcore-5.4.0 the dependency picks up M4 and binutils from dummy toolchain. I want to hide GCCcore and its deps, is this the correct way to do this. hpcswadm@hpcv18$eb GCCcore-5.4.0.eb -D --hidden --hide-deps=M4,Bison,flex,zlib,binutils == temporary log file in case of crash /tmp/eb-mZkRzA/easybuild-xv6sEZ.log Dry run: printing build status of easyconfigs and dependencies CFGS=/hpc/hpcswadm/easybuild * [ ] $CFGS/M4/M4-1.4.17.eb (module: Core | M4/.1.4.17) * [ ] $CFGS/Bison/Bison-3.0.4.eb (module: Core | Bison/.3.0.4) * [ ] $CFGS/flex/flex-2.6.0.eb (module: Core | flex/.2.6.0) * [ ] $CFGS/zlib/zlib-1.2.8.eb (module: Core | zlib/.1.2.8) * [ ] $CFGS/binutils/binutils-2.26.eb (module: Core | binutils/.2.26) * [ ] $CFGS/GCCcore/GCCcore-5.4.0.eb (module: Core | GCCcore/.5.4.0) Regards, Shahzeb Siddiqui HPC Linux Engineer B2220-447.2 Groton, CT
RE: [easybuild] GCCcore build in HMNS
Davide, That makes more sense now. Thanks for the clarification. I am having trouble figuring out the correct method for build GCC and hiding GCCcore module. I am trying to inspect the module file for GCC with extra dry run and it seems to be loading GCCcore/5.4.0 and not GCCcore/.5.4.0. I am passing –hide-toolchains=GCCcore and this is not what I expected Generating module file /nfs/grid/software/testing/RHEL7/easybuild/modules/all/Core/GCC/5.4.0-2.27.lua, with contents: help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/]]) whatis([[Description: The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/]]) local root = "/nfs/grid/software/testing/RHEL7/easybuild/software/Core/GCC/5.4.0-2.27" conflict("GCC") if not isloaded("GCCcore/5.4.0") then load("GCCcore/5.4.0") end if not isloaded("binutils/2.27") then load("binutils/2.27") end prepend_path("MODULEPATH", "/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/GCC/5.4.0-2.27") For some reason when I want to build GCC with –hide-toolchains it seems to rebuild GCCcore twice which is very inconvenient. It should only perform this action once but it is not, I wonder if anyone has been able to figure this out. hpcswadm@hpcv18$eb GCC-5.4.0-2.27.eb --robot -D --hide-toolchains=GCCcore == temporary log file in case of crash /tmp/eb-ZNSoSH/easybuild-DSa0KW.log Dry run: printing build status of easyconfigs and dependencies CFGS=/hpc/hpcswadm/easybuild * [x] $CFGS/M4/M4-1.4.17.eb (module: Core | M4/1.4.17) * [x] $CFGS/Bison/Bison-3.0.4.eb (module: Core | Bison/3.0.4) * [x] $CFGS/flex/flex-2.6.0.eb (module: Core | flex/2.6.0) * [x] $CFGS/zlib/zlib-1.2.8.eb (module: Core | zlib/1.2.8) * [x] $CFGS/binutils/binutils-2.27.eb (module: Core | binutils/2.27) * [ ] $CFGS/GCCcore/GCCcore-5.4.0.eb (module: Core | GCCcore/5.4.0) * [ ] $CFGS/GCCcore/GCCcore-5.4.0.eb (module: Core | GCCcore/.5.4.0) * [ ] $CFGS/M4/M4-1.4.17-GCCcore-5.4.0.eb (module: Compiler/GCCcore/5.4.0 | M4/1.4.17) * [ ] $CFGS/zlib/zlib-1.2.8-GCCcore-5.4.0.eb (module: Compiler/GCCcore/5.4.0 | zlib/1.2.8) * [ ] $CFGS/Bison/Bison-3.0.4-GCCcore-5.4.0.eb (module: Compiler/GCCcore/5.4.0 | Bison/3.0.4) * [ ] $CFGS/flex/flex-2.6.0-GCCcore-5.4.0.eb (module: Compiler/GCCcore/5.4.0 | flex/2.6.0) * [ ] $CFGS/binutils/binutils-2.27-GCCcore-5.4.0.eb (module: Compiler/GCCcore/5.4.0 | binutils/2.27) * [ ] $CFGS/GCC/GCC-5.4.0-2.27.eb (module: Core | GCC/5.4.0-2.27) Another issue I face is whenever I hide a module easybuild can’t find dependency for that particular module. I had this problem when I was building OpenMPI and built hwloc as hidden module, it couldn’t find it during the build. Is there something like –use-existing-module for hidden modules. I had been using filter-deps to bypass this issue, but it was not correct. Even though I might have had a os package to satisfy dependency some apps did not work correctly especially when they used –with-=$EBROOT From: easybuild-requ...@lists.ugent.be [mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Vanzo, Davide Sent: Wednesday, March 8, 2017 2:09 PM To: easybuild@lists.ugent.be Subject: Re: [easybuild] GCCcore build in HMNS Shahzeb, Looks good to me. It is expected that GCCcore depends on dummy toolchain built software since that is the only toolchain below GCCcore. In other words, you need whatever has been built with the system compiler and binutils in order to compile GCCcore. Once you have GCCcore built, at this point you will use it to compile again M4/Bison/flex/zlib/binutils within the GCCcore toolchain so that whatever is built with GCC is completely independent from the underlying OS software stack. -- Davide Vanzo, PhD Application Developer Adjunct Assistant Professor of Chemical and Biomolecular Engineering Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 www.accre.vanderbilt.edu<http://www.accre.vanderbilt.edu> On Mar 8 2017, at 12:58 pm, Siddiqui, Shahzeb mailto:shahzeb.siddi...@pfizer.com>> wrote: Hello, I want to find out if the build for GCCcore 5.4.0 is suppose to look like this under HMNS. I have copied all the files from easybuild repo into my local directory because I have disabled robot paths. One thing I don’t understand why there is M4 and binutils built with GCCcore toolchain while it also is a dependency in GCCcore. When I tried to build GCCcore-5.4.0 the dependency picks up M4 and binutils from dummy toolchain. I want to hide GCCcore and its deps, is this the correct way to do this.
Re: [easybuild] GCCcore build in HMNS
On 03/08/2017 01:08 PM, Vanzo, Davide wrote: Once you have GCCcore built, at this point you will use it to compile again M4/Bison/flex/zlib/binutils within the GCCcore toolchain so that whatever is built with GCC is completely independent from the underlying OS software stack. Which is still not entirely clear if always wanted. I might actually prefer to use the upstream vendor's version of those libraries. The current code is: class IccIfort(IntelIccIfort): """Compiler toolchain with Intel compilers (icc/ifort).""" NAME = 'iccifort' # use GCCcore as subtoolchain rather than GCC, since two 'real' compiler-only toolchains don't mix well, # in particular in a hierarchical module naming scheme SUBTOOLCHAIN = GCCcore.NAME Personally, using a flat MNS I'd be happy if SUBTOOLCHAIN name could be set to GCC instead of GCCcore when not using hierarchies But at present, if I want to share GCC(core) modules between intel/iomkl/foss, I have to build with GCCcore (and add binutils where needed). jack
Re: [easybuild] GCCcore build in HMNS
Shahzeb, Looks good to me. It is expected that GCCcore depends on dummy toolchain built software since that is the only toolchain below GCCcore. In other words, you need whatever has been built with the system compiler and binutils in order to compile GCCcore. Once you have GCCcore built, at this point you will use it to compile again M4/Bison/flex/zlib/binutils within the GCCcore toolchain so that whatever is built with GCC is completely independent from the underlying OS software stack. -- Davide Vanzo, PhD Application Developer Adjunct Assistant Professor of Chemical and Biomolecular Engineering Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 www.accre.vanderbilt.edu On Mar 8 2017, at 12:58 pm, Siddiqui, Shahzeb wrote: Hello, I want to find out if the build for GCCcore 5.4.0 is suppose to look like this under HMNS. I have copied all the files from easybuild repo into my local directory because I have disabled robot paths. One thing I don’t understand why there is M4 and binutils built with GCCcore toolchain while it also is a dependency in GCCcore. When I tried to build GCCcore-5.4.0 the dependency picks up M4 and binutils from dummy toolchain. I want to hide GCCcore and its deps, is this the correct way to do this. hpcswadm@hpcv18$eb GCCcore-5.4.0.eb -D --hidden --hide-deps=M4,Bison,flex,zlib,binutils == temporary log file in case of crash /tmp/eb-mZkRzA/easybuild-xv6sEZ.log Dry run: printing build status of easyconfigs and dependencies CFGS=/hpc/hpcswadm/easybuild * [ ] $CFGS/M4/M4-1.4.17.eb (module: Core | M4/.1.4.17) * [ ] $CFGS/Bison/Bison-3.0.4.eb (module: Core | Bison/.3.0.4) * [ ] $CFGS/flex/flex-2.6.0.eb (module: Core | flex/.2.6.0) * [ ] $CFGS/zlib/zlib-1.2.8.eb (module: Core | zlib/.1.2.8) * [ ] $CFGS/binutils/binutils-2.26.eb (module: Core | binutils/.2.26) * [ ] $CFGS/GCCcore/GCCcore-5.4.0.eb (module: Core | GCCcore/.5.4.0) Regards, Shahzeb Siddiqui HPC Linux Engineer B2220-447.2 Groton, CT
[easybuild] GCCcore build in HMNS
Hello, I want to find out if the build for GCCcore 5.4.0 is suppose to look like this under HMNS. I have copied all the files from easybuild repo into my local directory because I have disabled robot paths. One thing I don't understand why there is M4 and binutils built with GCCcore toolchain while it also is a dependency in GCCcore. When I tried to build GCCcore-5.4.0 the dependency picks up M4 and binutils from dummy toolchain. I want to hide GCCcore and its deps, is this the correct way to do this. hpcswadm@hpcv18$eb GCCcore-5.4.0.eb -D --hidden --hide-deps=M4,Bison,flex,zlib,binutils == temporary log file in case of crash /tmp/eb-mZkRzA/easybuild-xv6sEZ.log Dry run: printing build status of easyconfigs and dependencies CFGS=/hpc/hpcswadm/easybuild * [ ] $CFGS/M4/M4-1.4.17.eb (module: Core | M4/.1.4.17) * [ ] $CFGS/Bison/Bison-3.0.4.eb (module: Core | Bison/.3.0.4) * [ ] $CFGS/flex/flex-2.6.0.eb (module: Core | flex/.2.6.0) * [ ] $CFGS/zlib/zlib-1.2.8.eb (module: Core | zlib/.1.2.8) * [ ] $CFGS/binutils/binutils-2.26.eb (module: Core | binutils/.2.26) * [ ] $CFGS/GCCcore/GCCcore-5.4.0.eb (module: Core | GCCcore/.5.4.0) Regards, Shahzeb Siddiqui HPC Linux Engineer B2220-447.2 Groton, CT