At the Hutch we have stated to build all EB modules in containers. The container has the full stack of Lua, Lmod, Easybuild and our current Toolchain. The UID/GID of the container account matches our production system. And the build prefix used in the container matches our production environment. When working on a new package the container is started with docker run, the resulting software is installed inside the container which is ephemeral and does not touch production. If the build works correctly, the module is put into production, by running the container again with the volume mapped to our production environment. With the production software volume mapped to container the build prefix will write directly to our production volume.
I have a full working example: EBcb == EasyBuild Container Build https://github.com/FredHutch/EBcb John Dey On 1/17/19, 9:27 AM, "[email protected] on behalf of Josef Dvoracek" <[email protected] on behalf of [email protected]> wrote: hi Kenneth, > (and this is not EasyBuild's fault btw) +1 indeed! > Why are you taking this road exactly, what's the added benefit compared to installing directly into /sw It's fair to mention non-technical reasons too. I somehow like the idea of building software in containers since I was sysop on HPC system with Lustre FS. There the containerization was the easiest way to build kernel modules with/without OFED and with any kernel for any OS. (we had mix of centos, rhel, v6+v7). Since then I consider building in containers as convenient and safe approach and I was not thinking too much how to do it with Easybuild.. technical reasons: * reproducibility - On some systems there are devel RPMs just on login nodes, and on computes there are just the binary packages. Some software (X, VNC..) can be completely missing. It can lead to surprising results. (especially with not-so-well-tested, site-specific easyconfigs) Containerization also completely removes the possible human-factor when someone somewhere installs some nice RPM from $reasons.. Container has own recipe which is in git, so tracking of the build env is trivial. * independent build and deployment actions - yes, I could take one node and keep it drained in slurm till I'll check the build is fine.. One node is not too much, but let's do the things correctly.. Containerized build also mean that I can submit a job with the build task, and next day/week/month I have software tree ready somewhere in my $scratch and there is no need to take care about draining and later undraining the nodes.. (again, human factor elimination). btw. could you, please, share your workflow too? I'm quite curious, especially as I understood you do it in more simple way. cheers Josef Josef Dvoracek fzu.cz/~jose On 17. 01. 19 16:37, Kenneth Hoste wrote: > Dear Josef, > > On 17/01/2019 11:40, Josef Dvoracek wrote: >> hi, >> >> I am actually also looking forward for answers from more experienced >> users in this thread as the procedures around EB are something what I >> could definitely improve at my system. >> >> My findings after ~half year of using Easybuild (EB): >> >> * moving EB directories is bad as it always breaks something > > Indeed, don't assume you can do this (and this is not EasyBuild's > fault btw). > >> * you usually want the possibility of rebuilding/updating software >> tree and switching to the new one in some atomic operation, having >> possibility to rollback, especially if you're building own >> easyconfigs where you can expect new and fancy bugs >> * Update and rollback of software tree should be easy and fast >> operation. >> >> How I do it: >> >> As I want software build to be in /sw (EASYBUILD_PREFIX=/sw) I build >> in Singularity container with bind option "-B /scratch:/sw" - so the >> directory "/sw" inside container is mapped to "/scratch" outside. As >> I have also enough of RAM, I also do "-B /dev/shm:/builddir" - builds >> in tmpfs are IMO faster than on global cluster filesystem.. >> >> That means when build in container is done, and I like it, I have >> everything ready in /scratch and I can (move|rsync) it into /sw - for >> instance as a root job on idle nodes or something similar.. (yes, >> then you need to ensure your changes are just incremental). > > > This is an interesting approach, but it seems overkill to me... > > Why are you taking this road exactly, what's the added benefit > compared to installing directly into /sw ? > > > regards, > > Kenneth > >> >> cheers >> >> josef >> >> On 17. 01. 19 11:07, Loris Bennett wrote: >>> Hi, >>> >>> I set up EasyBuild in my own home directory and managed to build a >>> bunch >>> of software. I then copied the whole EasyBuild tree to a different >>> directory owned by a system user 'build'. >>> >>> Unfortunately the path to my home directory has been written to various >>> scripts and, I presume, may have been compiled into some binaries. >>> Thus >>> I just want to rebuild the software. >>> >>> However doing say >>> >>> eb GCCcore-7.3.0.eb --robot --rebuild >>> >>> fails with the following message to stdout: >>> >>> make: *** [all] Error 2 >>> (at >>> easybuild/software/EasyBuild/3.8.0/lib/python2.7/site-packages/easybuild_framework-3.8.0-py2.7.egg/easybuild/tools/run.py:501 >>> in parse_cmd_output) >>> == 2019-01-17 10:22:21,683 easyblock.py:2864 WARNING build failed >>> (first 300 chars): cmd " make -j 40 " exited with exit code 2 and >>> output: >>> make[1]: Entering directory >>> `/trinity/shared/easybuild/build/GCCcore/7.3.0/dummy-/gcc-7.3.0/stage1_obj' >>> >>> >>> and the following in the log file: >>> >>> == 2019-01-17 10:22:21,683 easyblock.py:2864 WARNING build failed >>> (first 300 chars): cmd " make -j 40 " exited with exit code 2 and >>> output: >>> make[1]: Entering directory >>> `/trinity/shared/easybuild/build/GCCcore/7.3.0/dummy-/gcc-7.3.0/stage1_obj' >>> >>> mkdir -p -- ./fixincludes >>> mkdir -p -- ./libiberty >>> mkdir -p -- ./lto-plugin >>> mkdir -p -- ./intl >>> mkdir -p -- ./gmp >>> mkdir -p -- ./libbacktrace >>> Co >>> == 2019-01-17 10:22:21,684 easyblock.py:286 INFO Closing log for >>> application name GCCcore version 7.3.0 >>> >>> As I am slightly pushed for time and probably have to recompile most >>> stuff anyway, I was wondering what is the most pragmatic way to >>> proceed. >>> >>> Of the directories >>> >>> apps/ >>> build/ >>> ebfiles_repo/ >>> moduleData/ >>> modules/ >>> software/ >>> sources/ >>> >>> could I just delete >>> >>> build/ >>> modules/ >>> software/ >>> >>> ? >>> >>> Cheers, >>> >>> Loris >>> >>
