Re: [easybuild] Software/easyconfigs update avail check
On 05/05/2017 08:16 AM, Henkel, Andreas wrote: > Hi, > > @Jens: Thank you very much for your detailed reply. I really appreciate it. > > Personally, I don't have a problem with all the easybuild packages since they > are not opening up for root. I rather appreciate that users could built up > their own eb on out clusterwide installation. And the time saving for > installation is great (except if you make a bad choice and choose foss-2017a > as default tool chain but now I know much more about eb because of all the > modifications I had to make ;-) ) > > There was another interesting point raised about update cycles: what if > something is misconfigured in the easyconfigs for blas for example which > would lead to wrong results for the users using that lib? I know that this is > not directly related to easybuild but wanted to ask anyway... > > About the update checker @Damian pointed to a related pull request (thank > you!). Do I get it right that there is no such thing as checking for newer > versions of installed software? The point why I'm asking is just because I'm > thinking about diving into this. Thinking about something similar to apt > update which simply lists available upgrades. Actually, we have some > python-code that does this for our installed modules and sends its results to > nagios/icinga. This would be more of an update notifier and don't do any > action. If there isn't something like that yet I can have a look how to adapt > our stuff to eb. Any pointer to a good place in the framework is appreciated. You could take the standpoint that if users don't request new versions, you won't build them. That's what we do for most of the codes. Some of the more heavily used ones, that we basically know we will get requests for anyhow, we try to stay ahead of the users on. But for the most part, if it's not requested, ignore it. -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90-580 14 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [easybuild] Software/easyconfigs update avail check
Hi, @Jens: Thank you very much for your detailed reply. I really appreciate it. Personally, I don't have a problem with all the easybuild packages since they are not opening up for root. I rather appreciate that users could built up their own eb on out clusterwide installation. And the time saving for installation is great (except if you make a bad choice and choose foss-2017a as default tool chain but now I know much more about eb because of all the modifications I had to make ;-) ) There was another interesting point raised about update cycles: what if something is misconfigured in the easyconfigs for blas for example which would lead to wrong results for the users using that lib? I know that this is not directly related to easybuild but wanted to ask anyway... About the update checker @Damian pointed to a related pull request (thank you!). Do I get it right that there is no such thing as checking for newer versions of installed software? The point why I'm asking is just because I'm thinking about diving into this. Thinking about something similar to apt update which simply lists available upgrades. Actually, we have some python-code that does this for our installed modules and sends its results to nagios/icinga. This would be more of an update notifier and don't do any action. If there isn't something like that yet I can have a look how to adapt our stuff to eb. Any pointer to a good place in the framework is appreciated. Best, Andreas
Re: [easybuild] eb in singularity
Hello Shahzeb, If you want to load /etc/profile each time you launch image, you could add -l to shebang (#!/bin/sh -l) in /exec /run /shell files located in container, it will simulate a shell login and will load /etc/profile Best regards, Cédric Clerget De: "Siddiqui, Shahzeb" À: easybuild@lists.ugent.be Cc: "Gregory M. Kurtzer" Envoyé: Jeudi 4 Mai 2017 18:53:35 Objet: [easybuild] eb in singularity Hello folks, I would like to find out how Easybuild and Singularity are going to work together. I am trying to design an eb environment in Singularity as a container solution to host all of the eb apps in a prod environment. Is there anyone in the HPC community that is working on this? Currently, I have setup a container environment that can build packages in container and also install RPMs from easybuild via Artifactory that is done through the bootstrap process. One of things that puzzles me is how to setup an isolated container environment that runs /etc/profile for container. I’ve noticed I need to do this inorder to get module command to work in container. Currently, I have to do this manually after shelling in. -bash-4.2$ singularity shell /nfs/grid/software/testing.img Singularity: Invoking an interactive shell within container... Singularity.testing.img> module --version sh: module: command not found Singularity.testing.img> env | grep MODULEPATH MODULEPATH_ROOT=/usr/share/modulefiles MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core Singularity.testing.img> unset MODULEPATH Singularity.testing.img> . /etc/profile Singularity.testing.img> . /nfs/grid/software/ module-setup.sh RHEL7/ RHEL7-BUILD/ Singularity.testing.img> . /nfs/grid/software/module-setup.sh Singularity.testing.img> ml av --- /usr/share/lmod/lmod/modulefiles/Core lmod/6.5.1 settarg/6.5.1 /nfs/grid/software/RHEL7-BUILD/easybuild/modules/all EasyBuild/3.1.2 /nfs/grid/software/RHEL7/easybuild/modules/all/Core - Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92 Java/1.8.0_92 gaussian/16-AVX iompi/2017.01 tbb/2017.2.132 GCC/5.4.0-2.27 Inspector/2017_update1 VTune/2017_update1 gaussian/16-SSE2 (D) ipp/2017.1.132 GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016 daal/2017.1.132 intel/2017.01 itac/2017.1.024 Where: D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". Singularity.testing.img> ml EasyBuild Singularity.testing.img> eb --version This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host amrndhl1157.pfizer.com. Singularity.testing.img> eb --show-config # # Current EasyBuild configuration # (C: command line argument, D: default value, E: environment variable, F: configuration file) # buildpath (D) = /home/siddis14/.local/easybuild/build installpath (D) = /home/siddis14/.local/easybuild repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo robot-paths (D) = /nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs sourcepath (D) = /home/siddis14/.local/easybuild/sources Singularity.testing.img> eb zlib-1.2.8.eb == temporary log file in case of crash /tmp/eb-BfnLvm/easybuild-ybJc4_.log == zlib/1.2.8 is already installed (module found), skipping == No easyconfigs left to be built. == Build succeeded for 0 out of 0 == Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have been removed. == Temporary directory /tmp/eb-BfnLvm has been removed. Singularity.testing.img> eb zlib-1.2.8.eb --rebuild == temporary log file in case of crash /tmp/eb-IMN3vl/easybuild-lWCjI1.log == processing EasyBuild easyconfig /nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb == building and installing zlib/1.2.8... == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... == postprocessing... == sanity checking... == cleaning up... == creat
[easybuild] eb in singularity
Hello folks, I would like to find out how Easybuild and Singularity are going to work together. I am trying to design an eb environment in Singularity as a container solution to host all of the eb apps in a prod environment. Is there anyone in the HPC community that is working on this? Currently, I have setup a container environment that can build packages in container and also install RPMs from easybuild via Artifactory that is done through the bootstrap process. One of things that puzzles me is how to setup an isolated container environment that runs /etc/profile for container. I've noticed I need to do this inorder to get module command to work in container. Currently, I have to do this manually after shelling in. -bash-4.2$ singularity shell /nfs/grid/software/testing.img Singularity: Invoking an interactive shell within container... Singularity.testing.img> module --version sh: module: command not found Singularity.testing.img> env | grep MODULEPATH MODULEPATH_ROOT=/usr/share/modulefiles MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core Singularity.testing.img> unset MODULEPATH Singularity.testing.img> . /etc/profile Singularity.testing.img> . /nfs/grid/software/ module-setup.sh RHEL7/ RHEL7-BUILD/ Singularity.testing.img> . /nfs/grid/software/module-setup.sh Singularity.testing.img> ml av --- /usr/share/lmod/lmod/modulefiles/Core lmod/6.5.1settarg/6.5.1 /nfs/grid/software/RHEL7-BUILD/easybuild/modules/all EasyBuild/3.1.2 /nfs/grid/software/RHEL7/easybuild/modules/all/Core - Advisor/2017_update1IGV/2.3.80-Java-1.8.0_92 Java/1.8.0_92 gaussian/16-AVX iompi/2017.01 tbb/2017.2.132 GCC/5.4.0-2.27 Inspector/2017_update1 VTune/2017_update1gaussian/16-SSE2 (D)ipp/2017.1.132 GCC/6.2.0-2.27 (D)IntelClusterChecker/2017.1.016 daal/2017.1.132 intel/2017.01 itac/2017.1.024 Where: D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". Singularity.testing.img> ml EasyBuild Singularity.testing.img> eb --version This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host amrndhl1157.pfizer.com. Singularity.testing.img> eb --show-config # # Current EasyBuild configuration # (C: command line argument, D: default value, E: environment variable, F: configuration file) # buildpath (D) = /home/siddis14/.local/easybuild/build installpath(D) = /home/siddis14/.local/easybuild repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo robot-paths(D) = /nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs sourcepath (D) = /home/siddis14/.local/easybuild/sources Singularity.testing.img> eb zlib-1.2.8.eb == temporary log file in case of crash /tmp/eb-BfnLvm/easybuild-ybJc4_.log == zlib/1.2.8 is already installed (module found), skipping == No easyconfigs left to be built. == Build succeeded for 0 out of 0 == Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have been removed. == Temporary directory /tmp/eb-BfnLvm has been removed. Singularity.testing.img> eb zlib-1.2.8.eb --rebuild == temporary log file in case of crash /tmp/eb-IMN3vl/easybuild-lWCjI1.log == processing EasyBuild easyconfig /nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb == building and installing zlib/1.2.8... == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... == postprocessing... == sanity checking... == cleaning up... == creating module... == permissions... == packaging... == COMPLETED: Installation ended successfully == Results of the build can be found in the log file(s) /home/siddis14/.local/easybuild/software/zlib/1.2.8/easybuild/easybuild-zlib-1.2.8-20170504.163248.log == Build succeeded for 1 out of 1 == Temporary log file(s) /tmp/eb-IMN3vl/easybuild-lWCjI1.log* have been removed. == Temporary directory /tmp/eb-IMN3vl has been removed.
Re: [easybuild] how to specify choices for dependencies?
Thanks, Alan and Gizo,
I'll take a look into this.
Would it be welcome to include this functionality in an upcoming PR to
include ecCodes and EMOS? Or rather not, since this apparently depends
on lmod being the module system of choice?
Cheers,
Andreas
Alan O'Cais writes:
> This sounds like a reasonable approach to me. You should probably specify one
> python as one of your `builddependencies` and then create a modluafooter that
> ensures you have a Python loaded, something like
> if not isloaded("Python") then
> load("Python")
> end
> That should still allow you to keep the python bindings as part of your
> sanity check too.
>
> Alan
>
> On 2 May 2017 at 21:12,
> mailto:nan...@luis.uni-hannover.de>> wrote:
>
> Hi,
>
> you could implemet this logic as some lua function defined in SitePackage.lua
> and then use the
> option modluafooter in easyconfig of emos. Not very nice, but should work.
>
> http://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html
>
> Best,
> Gizo
>
> Quoting Andreas Hilboll mailto:hilb...@uni-bremen.de>>:
>
> Hi all,
>
> I want to create easyconfigs for ecCodes[1] and EMOS[2]. ecCodes comes
> with Python bindings, so I'm setting a
>
> versionsuffix = '-Python-%(pyver)s'
>
> EMOS depends on ecCodes, but does not depend on Python at all.
>
> I have built two ecCodes modules, one for Python-2.7.11 and one for
> Python-3.5.1. I'm wondering how to specify the dependencies in the EMOS
> easyconfig, without having to provide a Python-dependent easyconfig for
> EMOS. Ideally, when loading EMOS, lmod should pick the ecCodes module
> which causes the least conflict:
>
> - if no other Python is loaded, chose any default
> - if either Python/2.7.11 or Python/3.5.1 are already loaded, chose the
> matching ecCodes module
>
> Is this somehow possible?
>
> Cheers,
> Andreas.
>
> [1] https://software.ecmwf.int/wiki/display/ECC
> [2] https://software.ecmwf.int/wiki/display/EMOS
>
> --
> Dr. Andreas Hilboll
>
> Center for Marine Environmental Sciences (MARUM)
> - AND -
> Institute of Environmental Physics (IUP)
>
> University of Bremen
>
> NW1 / S3132
> Otto-Hahn-Allee 1
> D-28359 Bremen
> Germany
>
> +49(0)421 218 62133 (phone)
> +49(0)421 218 98 62133 (fax)
> http://www.iup.uni-bremen.de/~hilboll
>
>
> --
> Dr. Gizo Nanava
> Leibniz Universitaet IT Services
> Leibniz Universitaet Hannover
> Schlosswender Str. 5
> D-30159 Hannover
> Tel +49 511 762 7919085
> http://www.luis.uni-hannover.de
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Software/easyconfigs update avail check
On 04-05-17 10:39, Jens Timmerman wrote: > > On 04/05/2017 08:31, Henkel, Andreas wrote: >> Hi, >> >> Recently we started using eb for our new cluster. Yesterday, in our group >> meeting a question was raised concerning updates and security patches for >> installed software similar to apt update/upgrade, yum update,...? >> Or is there a routine to check for newer releases of installed easyconfigs? > Short answer: No > > There are no such systems in place, EasyBuild does not have a security > team, so there are no systems in place for installing security updates. There is currently one exception: openssl. We have an easyconfig for it but we put it as a osdependency in all cases. So whatever security updates happen to the OS provided version of openssl, Easybuild uses it. Ward
Re: [easybuild] Software/easyconfigs update avail check
Just to add on that, you could create drop-in replacements for selected packages. You could create MariaDB-10.X-intel-2016b.eb (for instance), update the easyconfig file every time there is a new release in the 10.X series, and reinstall it on top of the old one, so bug fixes and security updates could be applied transparently* to the users and to other packages. That’s normally something you don’t want to do for reproducibility reasons, but it makes sense in certain cases. Damian *Static libraries might still be an issue though. On 04/05/17 10:39, "easybuild-requ...@lists.ugent.be on behalf of Jens Timmerman" wrote: I do realize there exist a few bad scenarios, e.g. software that opens up sockets to listen to incomming connections such as MariaDB and mongoDB. If these packages have security issues (or are badly configured) your users risk loss of confidentiality on their data or even arbitrary code executing as their user (not root, not a sudo user). Or a software package that has issues and opens up a users files to the world. This would require you to either educate your users not to use the bad software in an unsafe way, or manually remove it and install a patched version and tell them to use that version. If you are a non admin user using EasyBuild to install your own software on a system, you will need to follow up on this as there is currently nobody paid to do this centrally for all EB easyconfigs. Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt
Re: [easybuild] Software/easyconfigs update avail check
Hi Andreas On 04/05/2017 08:31, Henkel, Andreas wrote: > Hi, > > Recently we started using eb for our new cluster. Yesterday, in our group > meeting a question was raised concerning updates and security patches for > installed software similar to apt update/upgrade, yum update,...? > Or is there a routine to check for newer releases of installed easyconfigs? Short answer: No There are no such systems in place, EasyBuild does not have a security team, so there are no systems in place for installing security updates. The main point of EasyBuild is to get user software installed on the system, usable by users, for users. You would not run EasyBuild as root (this was even explicitly very hard to do until as of very recent, now it has a very clear config option informing you you should not use that option, but some people have their own reasons to do things as root) And you would not install software with setuid, and you would not run EasyBuild installed software as root (or more a privileged sudo capable user) but only as a non privileged user. As such from the point of a system administrator the security of your system would not be impacted by software installed by EasyBuild, at least, not any more then any software any user can scp to their home dir and run there. (At least, unless you mount home and scratch as no exec) Since we allow our users to write and compile their own software, EasyBuild was developed to provide centrally installed (and optimized) software as a convenience to our user, it has never been in our scope to have EasyBuild installed software to be the only allowable software to run. Furthermore, EasyBuild doesn't 'update' installed software. An installed software package will always stay installed, newer versions will be installed alongside it, not replace it, this is one of the main reasons to use EasyBuild, so you can easily use different versions of the same package on the same system. And lastly, the easyconfig files provided by EasyBuild should be seen as only 'examples'. People can privately have 1000's of applications that are not known and never published to the EasyBuild, so no central EasyBuild security team could ever notify you of updates for these packages. I do realize there exist a few bad scenarios, e.g. software that opens up sockets to listen to incomming connections such as MariaDB and mongoDB. If these packages have security issues (or are badly configured) your users risk loss of confidentiality on their data or even arbitrary code executing as their user (not root, not a sudo user). Or a software package that has issues and opens up a users files to the world. This would require you to either educate your users not to use the bad software in an unsafe way, or manually remove it and install a patched version and tell them to use that version. If you are a non admin user using EasyBuild to install your own software on a system, you will need to follow up on this as there is currently nobody paid to do this centrally for all EB easyconfigs. There are tools out there (like the fedora packagers tools) that will scan upstream links to automatically inform packagers of new installs. But as far as I know nobody has looked at this yet EasyBuild. Some work was started in to making it easy for you to bump all the dependencies of a given package to the latest version, [0] but this still relies in someone figuring out a new version is available, and adding it to the central (or local) Easyconfigs repository. I hope this clears up a few things for you. Regards, Jens Timmerman > Best, > Andreas Henkel [0] https://github.com/hpcugent/easybuild-framework/pull/2136
