Re: [easybuild] Beginner questions

[Fotis Georgatos]

Hi André,

On Mar 11, 2016, at 5:01 PM, André Gemünd  
wrote:
> Great. And what do people do when Intel releases Update 3 shortly after? As I 
> said, its probably more a question of best-practice. Do you skip these 
> versions? Or just update them without using EasyBuild? And if there is an 
> EasyConfig using an older release, do you stay with the one thats referenced 
> in the file for better reproducability (or some other reason), or simply 
> modify it?

The approach which I follow is to rely on buildsets (say, 
/opt/apps/HPCBIOS.20160312) and
simply invent a new buildsets when sufficient planets align (new configs, new 
version of EB etc).
More broadly, easyconfigs normally trickle up in the food chain like: 
user’s->group’s->system's
I think you question is more about how do you manage the system’s area, right?

Fotis

p.s. @ Kenneth:
I can see a more generic need here to swap our notes about what works, what 
doesn’t and how
in managing easyconfigs, so how about feeding this topic in one of our next 
30min hangouts?

-- 
echo "sysadmin know better bash than english" | sed s/min/mins/ \
  | sed 's/better bash/bash better/' # signal detected in a CERN forum

Re: [easybuild] Beginner questions

[Kenneth Hoste]

On 11/03/16 18:01, André Gemünd wrote:

Hi Kenneth,


But first of all: welcome to the wonderful world of EasyBuild! :-)

Thanks. :)


What you may be look for is configuration files, i.e.
http://easybuild.readthedocs.org/en/latest/Configuration.html#configuration-file-s
EasyBuild will pick up configuration files in /etc/easybuild.d/*.cfg, among
other locations, so maybe you want to specify configuration options specific to
the system there?

Yep, I've seen that. But what I'd actually would like to do is configure 
EasyBuild system-independent. :)
Its installed on a share, and I'd like to use a global configuration file.


Well, then you can make sure the exact same 
/etc/easybuild.d/defaults.cfg config file is available everywhere?




I tried to install intel-2015b.eb today, and found that it uses e.g. ccomp
2015.3.187, while intel already provides 2015.6.233. Now, should I copy the
existing easyconfig from folder A to folder B, edit it, then run the install?
For that, do I really need to search the
lib/python-xy/site-packages/xyz-xyz-py2.7-egg/easybuild/easyconfigs/letter/name/version
path, or is there some more comfortable way? It seems a bit cumbersome. Then,
is the robot path what I'm looking for for folder B? I'm a bit confused by the
name robot path, if its actually the easyconfig path?
Or do you guys work on a git clone of the easyconfig repo as robot path? I guess
this would make pull requests for changes a lot easier.

Different people have different approaches, you'll need to figure out what works
best for you.
Again, the documentation should help, cfr.:
http://easybuild.readthedocs.org/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path

Okay, although that doesn't really explain why the easyconfig path is called 
robot path?


That's a historical thing, it's just how we ended up naming it...

That's even before we started using .eb as extension for easyconfig 
files (or even before we started naming them as such...).




I guess I will try to point the robot path to a git clone of the latest 
easysconfig repository if that makes any sense.


It does make sense, if you want to pick up the latest stuff.

Be careful with following the 'develop' branch closely though, as new 
easyconfigs may require new features from the framework/easyblocks 
repository...


What most people do is maintain their own stack of easyconfig files on 
top of the ones shipped with EasyBuild, to deal with site-specific 
customizations and/or in-house easyconfigs.



Like Alan mentioned, we have more recent compositions of the Intel toolchain 
there, and
there's also two open pull requests for the very latest versions (2016 
update2), one on
top of GCC 4.9.3 (https://github.com/hpcugent/easybuild-easyconfigs/pull/2620), 
one on top
of GCC 5.3.0 (https://github.com/hpcugent/easybuild-easyconfigs/pull/2524).

Great. And what do people do when Intel releases Update 3 shortly after? As I 
said, its probably more a question of best-practice. Do you skip these 
versions? Or just update them without using EasyBuild? And if there is an 
EasyConfig using an older release, do you stay with the one thats referenced in 
the file for better reproducability (or some other reason), or simply modify it?


We don't modify the versions of dependencies in existing easyconfig 
files (unless there's a very good technical reason).


Someone looks into creating a new version of the required easyconfigs, 
and (hopefully) submits a pull request for them.
Then someone reviews/tests that PR, and it gets merged (into the 
'develop' branch), and included in the next EasyBuild release.



regards,

Kenneth

Re: [easybuild] Beginner questions

[Kenneth Hoste]

Hi André,

Let me pitch in too.

But first of all: welcome to the wonderful world of EasyBuild! :-)

On 10/03/16 13:29, André Gemünd wrote:

Hi EasyBuilders,

I'm new to EasyBuild and have a few questions. Let me say that maybe I'm just a 
bit confused here, so I apologize if these questions sound a bit daft. ;)

Regarding installation: is it possible to put a default configuration file in 
the installation tree of EasyBuild? We are several users that deploy software, 
and the systems used to build vary as well. I'd prefer not to configure these 
nodes differently for that or have different settings based on user accounts. 
I'm now setting INSTALLPATH_SOFTWARE, INSTALLPATH_MODULES, SOURCEPATH, etc. in 
a profile.d script, which I then need to deploy everywhere. I think this is 
needlessly complicated, if all I would need is for easybuild to parse a config 
file in its install path?
All details/options w.r.t. configuring EasyBuild are documented at 
http://easybuild.readthedocs.org/en/latest/Configuration.html .


What you may be look for is configuration files, i.e. 
http://easybuild.readthedocs.org/en/latest/Configuration.html#configuration-file-s 
. 

EasyBuild will pick up configuration files in /etc/easybuild.d/*.cfg, 
among other locations, so maybe you want to specify configuration 
options specific to the system there?



I tried to install intel-2015b.eb today, and found that it uses e.g. ccomp 
2015.3.187, while intel already provides 2015.6.233. Now, should I copy the 
existing easyconfig from folder A to folder B, edit it, then run the install? 
For that, do I really need to search the 
lib/python-xy/site-packages/xyz-xyz-py2.7-egg/easybuild/easyconfigs/letter/name/version
 path, or is there some more comfortable way? It seems a bit cumbersome. Then, 
is the robot path what I'm looking for for folder B? I'm a bit confused by the 
name robot path, if its actually the easyconfig path?
Or do you guys work on a git clone of the easyconfig repo as robot path? I 
guess this would make pull requests for changes a lot easier.


Different people have different approaches, you'll need to figure out 
what works best for you.



Again, the documentation should help, cfr.: 
http://easybuild.readthedocs.org/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path.


For the latest (typically already well tested) easyconfigs, see the 
'develop' branch in the GitHub repository, i.e. 
https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs 
. 

Like Alan mentioned, we have more recent compositions of the Intel 
toolchain there, and there's also two open pull requests for the very 
latest versions (2016 update2), one on top of GCC 4.9.3 
(https://github.com/hpcugent/easybuild-easyconfigs/pull/2620), one on 
top of GCC 5.3.0 
(https://github.com/hpcugent/easybuild-easyconfigs/pull/2524). Note: 
both of these use the newer/latest binutils 2.26.


I'm not sure yet if these will make it into the upcoming EasyBuild 
v2.7.0 release, I need to revisit them and maybe do some more testing 
beyond just HPL.



regards,

Kenneth

Re: [easybuild] Beginner questions

[André Gemünd]

Thanks for the input everyone! It will probably take me some time digesting and 
trying out these ideas.

Cheers
-- 
André Gemünd
Fraunhofer-Institute for Algorithms and Scientific Computing
andre.gemu...@scai.fraunhofer.de
Tel: +49 2241 14-2193
/C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

Re: [easybuild] Beginner questions

[André Gemünd]

Dear Alan, 

thanks for the feedback!

> We use a module file to control the settings you mention, it simply sets the
> equivalent EASYBUILD_* for the settings you require. Create the module and put
> it somewhere in accessible to the MODULEPATH (you may want a separate 
> protected
> directory for people who use this module).

Ah, yes, an obvious solution that I somehow didn't think about, because there 
already is an EasyBuild module created by the bootstrap script and I didn't 
want to modify that in case it would be modified in upgrades. But its a good 
idea, maybe a protected admin module.

> There's a later intel-2016b.eb already available which has a 2016 update and 
> an
> intel-2016.02 in the devel branch that has the very latest intel compilers. 

True, but we need to offer multiple versions at the same time, currently 
starting at XE 2012. Of the seperate point releases (2013, 2015, etc.), we'd 
like to offer the latest minor version, e.g. 2015 upd 6. 

> issue is that you will not find so many easyconfigs for these toolchains yet
> since they are relatively new. The ones you do find are the latest and 
> greatest
> however and you can add software from intel-2015b using
> --try-toolchain=intel,2016b (just beware of the notes in that link!).

Regarding applications its okay if we build them with the latest and greatest 
Intel toolchain. Thanks for the hint with try-toolchain, this seems very 
helpful! 

Greetings
-- 
André Gemünd
Fraunhofer-Institute for Algorithms and Scientific Computing
andre.gemu...@scai.fraunhofer.de
Tel: +49 2241 14-2193
/C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

Re: [easybuild] Beginner questions

[Fotis Georgatos]

Hi André,

On Mar 10, 2016, at 12:29 PM, André Gemünd  
wrote:
> Regarding installation: is it possible to put a default configuration file in 
> the installation tree of EasyBuild? We are several users that deploy 
> software, and the systems used to build vary as well. I'd prefer not to 
> configure these nodes differently for that or have different settings based 
> on user accounts. I'm now setting INSTALLPATH_SOFTWARE, INSTALLPATH_MODULES, 
> SOURCEPATH, etc. in a profile.d script, which I then need to deploy 
> everywhere. I think this is needlessly complicated, if all I would need is 
> for easybuild to parse a config file in its install path?

Considering EASYBUILD_CONFIGFILES?

If you go down the path of having a customised EasyBuild modulefile, you will 
eventually get in trouble 
by users who install newer EB versions (OK, they _should_ know better that it 
is it’s not too “easy" ;)

If you would hardwire variables under /etc/profile.d some things will work, 
yet shell inheritance would prevent free user customisation of these values :-( 
. 

There is a better way!

Relying upon Lmod, of Robert McLay’s fame ;)

So, an alternative approach might be to rely on default sticky Lmod modules, 
via LMOD_SYSTEM_DEFAULT_MODULES, fi. roughly as described in:
https://sourceforge.net/p/lmod/mailman/message/32783610/

The advantage of that technique is that it turns everything representable via 
variables
uniformly available across all users (who may still opt-out of it) at first 
shell invocation.
As a bonus, you’d now have a great mechanism to define $SCRATCH, $ARCHIVE etc.

And if you follow that thread carefully, you will also notice that the 
Lmod-based mechanism
even allows for restorable alias definitions (one more bonus - slash that 
.bashrc pollution!).

enjoy,
Fotis

-- 
echo "sysadmin know better bash than english" | sed s/min/mins/ \
  | sed 's/better bash/bash better/' # signal detected in a CERN forum