Re: [easybuild] Can one mix easybuild with other build tools?

[Jens Timmerman]

Hi payerle,

I believe this caveat means: If you have a naming scheme in place, and
start using a new one, EasyBuild will be confused and start overwriting
software made with the old module naming scheme.

e.g.
You install package foo-2.0.0 in  /software/Core/foo/2.0.0
You want to install bar which depends on foo-2.0.0
EasyBuild will attempt module load foo/2.0.0, find the foo module and just
start building bar

You now switch module naming scheme to also include the toolchain name in
the module name.
You want to install a new package bla with the foss toolchain and bla
 depends on foo:
EasyBuild will attempt module load foss/foo/2.0.0 and not find it, since
the module previously created is called foo/2.0.0
EasyBuild will attempt to install foo/2.0.0 but since you only changed the
module naming and not the install path, it will override
/software/Core/foo/2.0.0

This could also happen if you install packages manually, and not create
modules that have the exact name EasyBuild is looking for.

If you run reinstall --force EasyBuild will also override manually
installed packges, but that's what the --force is for, you should know what
you want when you use --force.

Hope this helps clarifying this a bit?

Regards,
Jens Timmerman

On 22 June 2018 at 18:26,  wrote:

> I am investigating the use of easybuild when we roll out our next cluster.
>
> One concern raised was from
> https://github.com/easybuilders/easybuild/wiki/
> Using-a-custom-module-naming-scheme
> where it clear states that different installation prefixes are needed for
> different easybuild naming schemes, or easybuild for schemeA might delete
> software built with schemeB.
>
> I am still getting familiar with easybuild, but this suggests to me that
> if I
> plan to have a mix of software built with easybuild and software built
> manually and/or with some other software build tools, that they would need
> to
> be installed in distinct software trees.  E.g., if I intend to build
> package
> foo with easybuild, putting it in /software/Core/foo/... and I intend to
> build
> package bar manually, then I better not put bar in /software/Core/bar but
> in
> something like /software2/Core/bar. (Using
> HierarchicalMNS naming scheme)
>
> Is my understanding of this correct?
> Does this only happen if rebuild/force flag is used?
> Does it only happen for the app trying to rebuild (and/or dependencies)?
> ---
> e.g. is it safe to have both foo and bar under /software/Core as long as I
> never issue an easybuild command that would try to rebuild bar (either
> directly or as a dependency)?  E.g., is the foo/bar example somewhat OK,
> but I
> only get burned if I e.g. built foo/1.2.3 with eb and foo/1.2.4 manually?
>
> Is there a list anywhere of the directories that EasyBuild wants full
> control
> over?   Is it the whole
> tree rooted at --prefix, or just the source, build, software install,
> module
> install paths?   to Would it be safe to have foo in
> /software/[Core|Compiler|
> MPI]/foo and bar directly under /software?
>

Re: [easybuild] eb in singularity

[Jens Timmerman]

Hi,

On 08/05/17 19:43, Di Pe wrote:

Interesting discussion about Singularity, there was an older thread here

https://mail.google.com/mail/u/0/#search/singularity/159045c3913ad572

This link doesn't work on my system.


and it would be great to learn what EB leadership thinks about 
integration of containers in EB?


To look a little bit into our requirements we would like to make it easy 
to share the software stacks we build using EB with


  1) other researchers outside our organization who are also running 
things on HPC systems
  2) other software platforms in the same organization (outside HPC 
clusters)
My answer here would be: share your EasyConfig files. If everything is 
setup correctly you will have an archive of installed EaysConfig files. 
You can share these with other people for use on their system. If you 
create 1 meta easyconfig that depends on all the software you need they 
can reproduce your environment with a single eb metaeasyconfig.eb --robot


In both cases we want to support a certain version of a containerized 
software stack for a certain period of time. 
What does a containerized software stack mean? Do you mean it is 
shielded form the other software on the system?

Take for example a data
science stack. We would bundle a certain version of R with a certain 
version of Python and some other ancillary tools. We would build a 
container that has one version of Python2 and one version of R and 
another container that has Python3 with R.

You can create 2 meta modules for these.
myfirstsmodule:
 - R 3.0
   - R 3.0 dependencies
 - Python 2.7
   - Python 2.7 dependencies
mysecondmodule:
 - R 3.1.0
  - R 3.1.0 dependencies
 - Python3
   - Python3 dependencies


I would not need or even want any Lmod environment inside the container 
because we may need to tie R and python together. For example R argparse 
is linked to a specific version of Python and Python rpy2 is linked to a 
specific version or R. This cannot be handled well with lmod.


Can you please explain a bit more what environment modules can't handle 
well here?

A case where 2 python modules would be loaded at the same time?




So perhaps it makes sense to focus on integrating Docker builds into EB 
as one could deliver a more comprehensive solution to bigdata, devops 
and HPC people.


Interested to hear some thoughts

The above being said, we do use singularity on our systems, and use 
Easybuild created software inside the containers. However, that would 
indeed require you to install lmod (or environment-modules) in your 
container.

We include in our /etc/singularity.conf

bind path = /apps
bind path = /etc/modulefiles # (Default location for environment-modules)

So all our EasyBuild built software is available inside the containers 
(and works if they have the correct glibc)

But this is probably the reverse of what you want to accomplish.

So for me the answer is:
If you want to distribute your software, distribute your Easyconfig 
files, and show the people how easily they can reproduce your build, 
should only be one command, if something goes wrong this should be 
considered a bug in EasyBuild (or a quirky thing in the softwares build 
procedure, which a clever person could fix in an EasyBlock)


EasyBuild will by default try to optimize for the hardware environment 
it was built on anyway, that is kind of the point of it, so creating a 
container and then sharing it and expecting the software to work on a 
different cpu is not something that's currently on our radar.
The entire point of EB was to have reproducible builds that non the less 
take advantage of your cpu architecture.


(these are my views at the moment, if someone wants to put a lot of work 
into extending EasyBuild to do different things than what was it's 
original point, I guess we would probably still merge it in, if it works 
and is useful to other people)



@Siddiqui, internally we have a wrapper around singularity that sets the 
environment and creates a list of commands that basically first invoke 
singularity and then run the profile and set other environment 
variables. It was in part (almost fully) written by Kenenth, so it 
integrates well with EasyBuild, I should inquire with Kenneth if we can 
make this public, or at least document what it does so you can benefit 
from his work :)


Regards,
Jens Timmerman


DP







On Thu, May 4, 2017 at 9:53 AM, Siddiqui, Shahzeb 
<shahzeb.siddi...@pfizer.com <mailto:shahzeb.siddi...@pfizer.com>> wrote:


Hello folks,

__ __

I would like to find out how Easybuild and Singularity are going to
work together. I am trying to design an eb environment in
Singularity as a container solution to host all of the eb apps in a
prod environment. Is there anyone in the HPC community that is
working on this?

__ __

Currently, I have setup a container environment that can build
packages in container and also install RPMs from easybuild via
   

Re: [easybuild] Software/easyconfigs update avail check

[Jens Timmerman]

Hi Andreas


On 04/05/2017 08:31, Henkel, Andreas wrote:
> Hi, 
>
> Recently we started using eb for our new cluster. Yesterday, in our group 
> meeting a question was raised concerning updates and security patches for 
> installed software similar to apt update/upgrade, yum update,...?
> Or is there a routine to check for newer releases of installed easyconfigs? 
Short answer: No

There are no such systems in place, EasyBuild does not have a security
team, so there are no systems in place for installing security updates.

The main point of EasyBuild is to get user software installed on the
system, usable by users, for users. You would not run EasyBuild as root
(this was even explicitly very hard to do until as of very recent, now
it has a very clear config option informing you you should not use that
option, but some people have their own reasons to do things as root) And
you would not install software with setuid, and you would not run
EasyBuild installed software as root (or more a privileged sudo capable
user) but only as a non privileged user.
As such from the point of a system administrator the security of your
system would not be impacted by software installed by EasyBuild, at
least, not any more then any software any user can scp to their home dir
and run there.
(At least, unless you mount home and scratch as no exec)
Since we allow our users to write and compile their own software,
EasyBuild was developed to provide centrally installed (and optimized)
software as a convenience to our user, it has never been in our scope to
have EasyBuild installed software to be the only allowable software to run.

Furthermore, EasyBuild doesn't 'update' installed software. An installed
software package will always stay installed, newer versions will be
installed alongside it, not replace it, this is one of the main reasons
to use EasyBuild, so you can easily use different versions of the same
package on the same system.

And lastly, the easyconfig files provided by EasyBuild should be seen as
only 'examples'. People can privately have 1000's of applications that
are not known and never published to the EasyBuild, so no central
EasyBuild security team could ever notify you of updates for these packages.

I do realize there exist a few bad scenarios, e.g. software that opens
up sockets to listen to incomming connections such as MariaDB and
mongoDB. If these packages have security issues (or are badly
configured) your users risk loss of confidentiality on their data or
even arbitrary code executing as their user (not root, not a sudo user).
Or a software package that has issues and opens up a users files to the
world.
This would require you to either educate your users not to use the bad
software in an unsafe way, or manually remove it and install a patched
version and tell them to use that version. If you are a non admin user
using EasyBuild to install your own software on a system, you will need
to follow up on this as there is currently  nobody paid to do this
centrally for all EB easyconfigs.

There are tools out there (like the fedora packagers tools) that will
scan upstream links to automatically inform packagers of new installs.
But as far as I know nobody has looked at this yet EasyBuild.
Some work was started in to making it easy for you to bump all the
dependencies of a given package to the latest version, [0] but this
still relies in someone figuring out a new version is available, and
adding it to the central (or local) Easyconfigs repository.

I hope this clears up a few things for you.

Regards,
Jens Timmerman
> Best, 
> Andreas Henkel
[0] https://github.com/hpcugent/easybuild-framework/pull/2136

Re: [easybuild] toolchain which uses existing, optimized, MPI/ScaLAPACK and build environment?

[Jens Timmerman]

Hello Gunnar,


On 24/01/2017 19:54, Gunnar Sauer wrote:
> Hello EasyBuild experts,
>

> But which toolchain do I choose on the Xeon cluster, which provides
> all those optimized tools through already existing modules? Can I
> tweak the goolf toolchain to use the existing system modules?
Yes, you could create your own toolchain to use the already existing
modules, this is exactly how the Cray toolchain works, see
http://easybuild.readthedocs.io/en/latest/Using_external_modules.html
for more information on how to create your own toolchain from existing
compilers and libraries.
> Or do I have to use the dummy toolchain instead? If using the dummy
> toolchain, do I have to rewrite all involved easyconfig files, or can
> I take - say the easyconfigs for the goolf toolchain - and somehow
> replace goolf by dummy at buildtime?
This is also an option, you can use the `--try-toolchain=dummy,dummy`
option, see
http://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html?#tweaking-existing-easyconfig-files-using-try
But I don't really recommended this, the first option is probably better.
> I hope, I could make my problem clear. Please help an EasyBuild newbie :)
The foss toolchain (goolf is an older toolchain) is indeed not the 'most
optimized' compiler toolchain, but it is optimized for the systems it
runs on best we can currently do with free and open source software.
We also support other toolchains with e.g.  Intel, pgi or CUDA compilers
And yes, these toolchains have infiniband support.

So, it would be very nice to know what optimizations are being done at
your company that make the internal toolchain even better optimized, so
all EasyBuild
users could all benefit from this knowledge and potentially millions of
CPU hours could be saved.
I'm really serious here, if you can share this information, we would
love to hear it so we can incorporate, but I do understand that this
might be proprietary information.

TL;DR:
If you can share your highly optimized toolchains with us we will be
pleased to support them in EasyBuild if they can help us getting faster
software runtimes!

Regards,
Jens Timmerman
>
> Thank you
> Gunnar

Re: [easybuild] R extra packages

[Jens Timmerman]

On 17/12/2016 04:54, Jack Perdue wrote:

On 12/16/2016 04:37 PM, Jens Timmerman wrote:

Hi,


On 16/12/2016 20:44, Siddiqui, Shahzeb wrote:
It comes down to whether we want simplicity in R, Python, or Perl 
installation by having packages installed inside the easyconfig 
file. It works well for most packages assuming there is no issue 
during build. There are cases when some package don't build properly 
and having to troubleshoot them with an entire list of packages 
become difficult.


I would like to decouple the R packages into individual easyconfig 
file, because I can resolve any dependencies and work on installing 
the particular package. Even if the process might be repetitive for 
most builds, at least the R collection will be built correctly. 
Another benefit is that I would like to package into RPMs, and it 
would be beneficial to create a RPM for each R package rather than 
just the main R. Just like how RHEL provides Python RPMs for its 
base and Python related packages, I think we can achieve this with 
Easybuild.


I also need packages from Bioconductor and would like this in 
Easybuild.



You can already do this, just use the RPackage easyblock: 
https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/rpackage.py

see for example all these easyconfigs:
https://github.com/hpcugent/easybuild-easyconfigs/search?utf8=%E2%9C%93=rpackage 



As you will see these already use packages from Bioconductor etc.

In fact this was the first way to install R packages in EasyBuild, 
the ext_lists option was only added later.
This Rpackage EasyBlock will install the R package like a user would 
normally do, and generate a module with an extra R_LIBS environment 
variable so R will be able to find it.



Regards,
Jens Timmerman


Yeah... but there's a lot to be said for a) breaking it down to
individual blocks with known version numbers (for tracking) so you can 
debug

them individually
Yes, that's exactly what is happening here. See the lists of easyconfigs 
I link to, a lot of seperate ones for just one R package

and b) avoiding having to wait an hour on a
28-core system for your EB-driven R build to fail because some extension
had a problem.




As for having your R build fail beacause 1 extension has a problem, I 
always worked around this by first doing a `bare` R install
which would generate a working R module, and then add the ext_list to 
the eb and run it again with `--keep`


This would keep the current R build and install the R extensions, if one 
failed to install the previously installed ones would still be there.



Regards,
Jens

For our R person here (who supports the R users), I install a fairly bare
version that allows for PIC offloading (via MKL) and he does all the 
extensions.


As for PythonPackage installing .eggs from other packages without
warning me, I'm going to have to talk to somebody.   I don't like it.

jack


-Original Message-
From: easybuild-requ...@lists.ugent.be 
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue

Sent: Friday, December 16, 2016 1:38 PM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] R extra packages

On 12/16/2016 10:54 AM, Siddiqui, Shahzeb wrote:

Hi Jack,

I checked out your work, this is great work. Did you consider 
sending your easyconfig files to EB. I think your collection of 
easyconfig files is a good starting point for building out the R 
collection. I think it is imperative to have R collection by 
version because these R packages change along with the version of R.


I would like if we can make a R collection for R 3.3.1 which is the 
latest one.
If anything, I would send the script(s) to EB...  I wouldn't want to 
swamp them with any more easyconfigs than I haven't already swamped 
them with (got lots that I need to send in).


But I would want to do Python and Perl too.  And I'd want support 
for things like Bioconducter and whoever.


It wasn't real popular here but we tend to make things easy for 
users (to not confuse them).
Making it easy for ourselves seems to be another argument (hard to 
disagree with the ease of "pip install").


But if I was doing performance analysis like I've done in the past 
I'd want to nail down every version I'm using and make sure that 
some random update didn't warp my data.


jack



-Original Message-
From: easybuild-requ...@lists.ugent.be
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue
Sent: Friday, December 16, 2016 11:39 AM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] R extra packages

On 12/16/2016 10:22 AM, Siddiqui, Shahzeb wrote:

Hello,

I would like to create a easyconfig for each R package, I find that
to be best suitable for our environment at Pfizer. I know we can pass
along the R packages in the R easybuild using exts_list . The only
issue is we have our own set of R packages that we need to install,
and I am not sure in which order they need to be installed.

It's important t

Re: [easybuild] R extra packages

[Jens Timmerman]

Hi,


On 16/12/2016 20:44, Siddiqui, Shahzeb wrote:

It comes down to whether we want simplicity in R, Python, or Perl installation 
by having packages installed inside the easyconfig file. It works well for most 
packages assuming there is no issue during build. There are cases when some 
package don't build properly and having to troubleshoot them with an entire 
list of packages become difficult.

I would like to decouple the R packages into individual easyconfig file, 
because I can resolve any dependencies and work on installing the particular 
package. Even if the process might be repetitive for most builds, at least the 
R collection will be built correctly. Another benefit is that I would like to 
package into RPMs, and it would be beneficial to create a RPM for each R 
package rather than just the main R. Just like how RHEL provides Python RPMs 
for its base and Python related packages, I think we can achieve this with 
Easybuild.

I also need packages from Bioconductor and would like this in Easybuild.


You can already do this, just use the RPackage easyblock: 
https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/rpackage.py

see for example all these easyconfigs:
https://github.com/hpcugent/easybuild-easyconfigs/search?utf8=%E2%9C%93=rpackage

As you will see these already use packages from Bioconductor etc.

In fact this was the first way to install R packages in EasyBuild, the 
ext_lists option was only added later.
This Rpackage EasyBlock will install the R package like a user would 
normally do, and generate a module with an extra R_LIBS environment 
variable so R will be able to find it.



Regards,
Jens Timmerman

-Original Message-
From: easybuild-requ...@lists.ugent.be 
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue
Sent: Friday, December 16, 2016 1:38 PM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] R extra packages

On 12/16/2016 10:54 AM, Siddiqui, Shahzeb wrote:

Hi Jack,

I checked out your work, this is great work. Did you consider sending your 
easyconfig files to EB. I think your collection of easyconfig files is a good 
starting point for building out the R collection. I think it is imperative to 
have R collection by version because these R packages change along with the 
version of R.

I would like if we can make a R collection for R 3.3.1 which is the latest one.

If anything, I would send the script(s) to EB...  I wouldn't want to swamp them 
with any more easyconfigs than I haven't already swamped them with (got lots 
that I need to send in).

But I would want to do Python and Perl too.  And I'd want support for things 
like Bioconducter and whoever.

It wasn't real popular here but we tend to make things easy for users (to not 
confuse them).
Making it easy for ourselves seems to be another argument (hard to disagree with the ease 
of "pip install").

But if I was doing performance analysis like I've done in the past I'd want to 
nail down every version I'm using and make sure that some random update didn't 
warp my data.

jack



-Original Message-
From: easybuild-requ...@lists.ugent.be
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue
Sent: Friday, December 16, 2016 11:39 AM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] R extra packages

On 12/16/2016 10:22 AM, Siddiqui, Shahzeb wrote:

Hello,

I would like to create a easyconfig for each R package, I find that
to be best suitable for our environment at Pfizer. I know we can pass
along the R packages in the R easybuild using exts_list . The only
issue is we have our own set of R packages that we need to install,
and I am not sure in which order they need to be installed.

It's important to separate the R installation from the R packages
because in the case of a build error in R package it would fail to
build R if they are both together.

I would like to build R packages as modules which will be
particularly important for some special R packages that require a
special dependency. For instance we need R2Jags and that has a
dependency for JAGS. I have JAGS in a module environment so I would
like R2Jags module file to load Jags before a user can use R2Jags.

When loading base R module it will allow user to have access to base
R packages that is quite small. The special R packages that are build
with Easybuild will have their R module only visible when loading R.
This can be done via MODULEPATH variable.

Similar to how you have Python package like numpy, I would like this
framework for R packages.


If you want to look over my experiment for automatically building easyconfigs 
based on CRAN data, see:

 http://www.siliconslick.com/easybuild/R_madness/

There are the scripts (which will need work), the easyconfigs created, and the 
modules built from those.

I thought it was a great idea.  Others on our staff, not so much (they are 
content to use R and Python's built in tools for installing to a dir

Re: [easybuild] R extra packages

[Jens Timmerman]

Hi,


On 16/12/2016 20:44, Siddiqui, Shahzeb wrote:

It comes down to whether we want simplicity in R, Python, or Perl installation 
by having packages installed inside the easyconfig file. It works well for most 
packages assuming there is no issue during build. There are cases when some 
package don't build properly and having to troubleshoot them with an entire 
list of packages become difficult.

I would like to decouple the R packages into individual easyconfig file, 
because I can resolve any dependencies and work on installing the particular 
package. Even if the process might be repetitive for most builds, at least the 
R collection will be built correctly. Another benefit is that I would like to 
package into RPMs, and it would be beneficial to create a RPM for each R 
package rather than just the main R. Just like how RHEL provides Python RPMs 
for its base and Python related packages, I think we can achieve this with 
Easybuild.

I also need packages from Bioconductor and would like this in Easybuild.


You can already do this, just use the RPackage easyblock: 
https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/rpackage.py

see for example all these easyconfigs:
https://github.com/hpcugent/easybuild-easyconfigs/search?utf8=%E2%9C%93=rpackage

As you will see these already use packages from Bioconductor etc.

In fact this was the first way to install R packages in EasyBuild, the 
ext_lists option was only added later.
This Rpackage EasyBlock will install the R package like a user would 
normally do, and generate a module with an extra R_LIBS environment 
variable so R will be able to find it.



Regards,
Jens Timmerman

-Original Message-
From: easybuild-requ...@lists.ugent.be 
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue
Sent: Friday, December 16, 2016 1:38 PM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] R extra packages

On 12/16/2016 10:54 AM, Siddiqui, Shahzeb wrote:

Hi Jack,

I checked out your work, this is great work. Did you consider sending your 
easyconfig files to EB. I think your collection of easyconfig files is a good 
starting point for building out the R collection. I think it is imperative to 
have R collection by version because these R packages change along with the 
version of R.

I would like if we can make a R collection for R 3.3.1 which is the latest one.

If anything, I would send the script(s) to EB...  I wouldn't want to swamp them 
with any more easyconfigs than I haven't already swamped them with (got lots 
that I need to send in).

But I would want to do Python and Perl too.  And I'd want support for things 
like Bioconducter and whoever.

It wasn't real popular here but we tend to make things easy for users (to not 
confuse them).
Making it easy for ourselves seems to be another argument (hard to disagree with the ease 
of "pip install").

But if I was doing performance analysis like I've done in the past I'd want to 
nail down every version I'm using and make sure that some random update didn't 
warp my data.

jack



-Original Message-
From: easybuild-requ...@lists.ugent.be
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue
Sent: Friday, December 16, 2016 11:39 AM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] R extra packages

On 12/16/2016 10:22 AM, Siddiqui, Shahzeb wrote:

Hello,

I would like to create a easyconfig for each R package, I find that
to be best suitable for our environment at Pfizer. I know we can pass
along the R packages in the R easybuild using exts_list . The only
issue is we have our own set of R packages that we need to install,
and I am not sure in which order they need to be installed.

It's important to separate the R installation from the R packages
because in the case of a build error in R package it would fail to
build R if they are both together.

I would like to build R packages as modules which will be
particularly important for some special R packages that require a
special dependency. For instance we need R2Jags and that has a
dependency for JAGS. I have JAGS in a module environment so I would
like R2Jags module file to load Jags before a user can use R2Jags.

When loading base R module it will allow user to have access to base
R packages that is quite small. The special R packages that are build
with Easybuild will have their R module only visible when loading R.
This can be done via MODULEPATH variable.

Similar to how you have Python package like numpy, I would like this
framework for R packages.


If you want to look over my experiment for automatically building easyconfigs 
based on CRAN data, see:

 http://www.siliconslick.com/easybuild/R_madness/

There are the scripts (which will need work), the easyconfigs created, and the 
modules built from those.

I thought it was a great idea.  Others on our staff, not so much (they are 
content to use R and Python's built in tools for installing to a dir

Re: [easybuild] Has anyone created EB files for the Intel 2017 compilers?

[Jens Timmerman]

On 03/11/2016 12:07, Fotis Georgatos wrote:

Hi Ole, all,

On Nov 2, 2016, at 2:33 PM, Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk> wrote:

On 11/02/2016 12:52 PM, Åke Sandgren wrote:

env INTEL_LICENSE_FILE=port@host eb intel-2016a.eb ...

Fantastic, this worked for me!  I wonder where the usage of license_file and/or 
INTEL_LICENSE_FILE within EB might be documented?

It SHOULD ;-)

There’s one caveat with the port@host approach:

One day, Intel’s practices will force you to setup a second license server
(either because you need to setup a new flexlm server version or,
because Intel has the nasty habit of silently deprecating old components in new 
licenses
and you need to point to a test instance without touching your production 
instance yet)
IMHO it's a good practice to let license server's hostnames be a cname 
that points to an A record, you can then still easily switch servers if 
you need to by moving the cname to point to a different A record. This 
also makes it easier for your users, they don't need to


Regards,
Jens Timmerman

Fotis

Re: [easybuild] Gmane is gone...what next?

[Jens Timmerman]

Markus,


On 29/09/2016 16:08, Markus Geimer wrote:

Jens,


It looks like the messages from August this year haven't been
uploaded (or somehow got lost otherwise)...  Any chance to upload
those too?

This has now been done,
and I also noticed that google has indexed the list on the mail archive, 
so now EasyBuild related searches sometimes include messages from the 
mailing list archives.


Regards,
Jens Timmerman


Thanks,
Markus

--
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone:  +49-2461-61-1773
Fax:+49-2461-61-6656
E-mail: m.gei...@fz-juelich.de
WWW:http://www.fz-juelich.de/jsc/





Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

[easybuild] Python 2.6 usage poll

[Jens Timmerman]

Hi all, 

In our organization we're planning on upgrading all our oldest systems to SL7 
by the end of the year.
This would mean we no longer have systems which have python 2.6 as their system 
python.

This means that for our internal tools we can drop support for python2.6. This 
also makes it easier 
to write code that can work in python3 (perhaps after it has gone trough the 
py2to3 tool)

Now I am curious what system python is being used in other HPC institutes, so I 
have set up a 
doodle where you can let me know your python versions/plans.

If you would be so kind to fill this in it would give me a sense of what is 
going on in the 
python 2.x/3.x support landscape.

As a note: RHEL 5 is python 2.4, RHEL 5 is python 2.6, RHEL 7 is python 2.7

http://doodle.com/poll/ir67d2hfhkefxfcp#table


Regards,
Jens Timmerman

Ps: as you might know EasyBuild dropped support to run on python 2.4 in version 
2.0
There aren't any plans on dropping 2.6, but this poll might give us some 
insights.

Re: [easybuild] Gmane is gone...what next?

[Jens Timmerman]

Hi all, 


On 09/01/2016 04:39 PM, Kenneth Hoste wrote:
> 
> 
> 
> Yep! https://www.mail-archive.com/easybuild@lists.ugent.be/
> 
> Thanks for looking into this Jens!
> 
> So, next up is uploading the full archive?
> 
full archive has been uploaded and is available for search on the mail-archive, 
I hope google index will also catch up to it soon.

https://www.mail-archive.com/easybuild@lists.ugent.be/

Regards,
Jens Timmerman

> 
> regards,
> 
> Kenneth

Re: [easybuild] Google Searchable Mailing List

[Jens Timmerman]

Hi Martin, 



On 07/26/2016 01:31 PM, Martin wrote:
> Hi,
> 
> Is there a way to easily find things via Google for the EB mailing lists?

This question has been asked here before, I have enabled indexing of the 
mailing list in gmane.org, 
see http://blog.gmane.org/gmane.comp.package-management.easybuild/

This way the list is indexed by google, but email addresses remain hidden from 
spam bots.

I believe it only started indexing since somewhere in Januari 2016, I have not 
yet uploaded the full maillist archive, 
I'll look into this.

Regards,
Jens Timmerman

> 
> It looks like none of the archives are indexed and I'm getting a 500 when 
> trying to use the search field from https://lists.ugent.be/wws.
> 
> /Martin
> -- 
> -- 
> http://www.xing.com/profile/Martin_Marcher
> http://www.linkedin.com/in/martinmarcher
> Mobil: +43 / 660 / 62 45 103
> UID: ATU68801424

Re: [easybuild] Quck question? Plain text "source" files (no decompression needed)

[Jens Timmerman]

Hi Jack, 

You can override the default 'unpack command' by specifying a tuple of 
('source', 'unpack command') to your sources.
e.g. how this is done here
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/a/ABINIT/ABINIT-7.0.3-x86_64_linux_gnu4.5.eb

I believe you could supply
sources = [SOURCE_TAR_GZ, ('Makefile.def', 'echo %s')]  # echo is a no-op

Regards,
Jens Timmerman

On 07/21/2016 05:28 PM, Jack Perdue wrote:
> Howdy all,
> 
> Is there an easy way to tell EB to not try to
> decompress source=[] files?  e.g. plain-text Makefile
> 
> extract_file seems to have issues and I"m forced
> to gunzip/gzip the Makefile.def I want to use for OpenSees
> everytime I need to tweak it.  e.g.
> 
> sources = [SOURCE_TAR_GZ, 'Makefile.def.gz']
> 
> I looked over the "Writing EasyConfigs: The Basics" and ReadTheDocs
> and didn't see what I need.  I also didn't see the "Advanced" page.
> 
> Looking through the framework I'm sure there is a simple
> solution, I'm just not seeing it at the moment.
> 
> Help?
> 
> Thanks,
> 

Re: [easybuild] Sourceforge download

[Jens Timmerman]

Hi Niek,

SOURCELOWER_TAR_BZ2 results in 'bridger-r2014.12.01'
so this wil  not work in the sources field.

your version seems wrong aswell
This seems to work for me:


easyblock = 'MakeCp'

name = 'Bridger'
version = 'r2014-12-01'

homepage = 'https://wiki.gacrc.uga.edu/wiki/Bridger'
description = """Bridger is an efficient de novo trascriptome assembler 
for RNA-Seq data. """


toolchain = {'name': 'goolf', 'version': '1.7.20'}
toolchainopts = {'pic': True, 'usempi': True}


sources = ['%(name)s_%(version)s.tar.gz']
source_urls = [('http://downloads.sourceforge.net/project/rnaseqassembly')]


files_to_copy = [('Assemble','bin/')]

moduleclass = 'bio'

On 24/09/15 10:55, Niek de Klein wrote:

Hi easybuild list,

I'm trying to install Bridger from sourceforge. The sourceforge link is:

sourceforge.net/projects/rnaseqassembly/files/Bridger_r2014-12-01.tar.gz/download 





So I tried using

name = 'Bridger'
version = 'r2014.12.01'
[..]
source_urls = [SOURCEFORGE_SOURCE]
sources = ['%%(name)s_%s.tar.gz' % '-'.join(version.split('.'))]

but this does not find the file to download. Looking at other programs 
installed from sourceforge it looks for this


sourceforge.net/projects/rnaseqassembly/files/bridger//2014.12.01/Bridger_2014-12-01.tar.gz/download 



so I think because this does not make use of 
// SOURCEFORGE_SOURCE won't work, 
correct?


So then I tried with

source_urls = ['sourceforge.net/projects/rnaseqassembly/files/ 
']
 sources = ['%%(name)s_%s.tar.gz/download' % 
'-'.join(version.split('.'))]


But this gives the error
 Unknown file type for file 
/apps/sources/b/Bridger/Bridger_r2014-12-01.tar.gz/download 
(['download'])")


Without /download at the end it doesn't find the url.

Other things I have tried:

   source_urls = ['sourceforge.net/projects/rnaseqassembly/files/ 
']
   sources = [('%%(name)s_%s.tar.gz' % 
'-'.join(version.split('.')),'download')]



  sources = [SOURCELOWER_TAR_BZ2]
  source_urls = 
[('http://sourceforge.net/projects/rnaseqassembly/files/', 'download')]


and

  sources = [SOURCELOWER_TAR_BZ2]
  source_urls = [('http://sourceforge.net/projects/rnaseqassembly/', 
'download')]


But then the files can't be found. I don't know what else to try, how 
can I get this downloaded from sourceforge? Below the full .eb file 
for replication:


easyblock = 'MakeCp'

name = 'Bridger'
version = 'r2014.12.01'

homepage = 'https://wiki.gacrc.uga.edu/wiki/Bridger'
description = """Bridger is an efficient de novo trascriptome 
assembler for RNA-Seq data. """


toolchain = {'name': 'goolf', 'version': '1.7.20'}
toolchainopts = {'pic': True, 'usempi': True}


sources = [SOURCELOWER_TAR_BZ2]
source_urls = 
[('http://sourceforge.net/projects/rnaseqassembly/files', 'download')]



files_to_copy = [('Assemble','bin/')]

moduleclass = 'bio'

Re: [easybuild] openssl-devel, zlib-devel and EB

[Jens Timmerman]

Hi Jack,

On 19/06/15 19:05, Jack Perdue wrote:

Howdy all,




on RHEL, openssl-devel requires zlib-devel.
And unless great care is taking during the configure_step,
most packages will invariably pick up the version in /usr
and not the one at $EBROOTZLIB.



What would be a better one?

lately I've been thinking that we should maybe have an openssl-devel 
module in EasyBuild.
This would be a openssl module that only ships the header files, so you 
don't need to install openssl-devel on your system.

However, it would still link with the openssl libraries on your system.

Currently EasyBuild doesn't do like most distributions, that split up 
header files into these -devel modules, but puts everything in


So if you update your system's libraries you still have the benefits,
if the ".so" compatibility breaks your software compiled against it will 
break, but you already have this case when you
link against the system libraries now, and maybe this is what you want, 
so insecure applications stop working automatically.
(but I don't think RHEL ever puts out updates that break ".so" 
compatibility?


I'm not sure if this is better, but it's an idea I wanted to throw out 
there.


jack

Regards,
Jens Timmerman

Re: [easybuild] Script folder missing

[Jens Timmerman]

Hi,


On 12/06/15 10:14, Valvanuz Fernández wrote:


The scripts folder is not present. Is this scripts folder important? 
Should I download it from the EasyBuild website?


The scripts folder is not needed for day to day use of EasyBuild, it 
contains some useful scripts that can help you
convert certain easyconfigs, generate lists, generate dependencies for 
perl modules...


So it might be interesting to have a look at what scripts we do have, 
but they are only there for your convenience, not

because they are needed.


Regards,
Jens Timmerman

Thank you in advanced

Valvanuz

Re: [easybuild] Include toolchain dependency in module name or not?

[Jens Timmerman]

Hi Franky,


On 03/06/15 14:18, Backeljauw Franky wrote:

Hello,

We have a new EasyConfig file SOBAcl-r18.eb which has this in it:

easyblock = "Tarball"

name = 'SOBAcl'
version = 'r18'

toolchain = {'name': 'dummy', 'version': 'dummy'}

dependencies = [('GraphViz', '2.18-intel-2014b-Perl-5.20.0')]




if you set the toolchain to intel-2015a change this to

dependencies = [('GraphViz', '2.18', '-Perl-5.20.0', True)]

True means, use the toolchain that's used for this easyconfig.
or if you keep dummy dummy you can specify:

dependencies = [('GraphViz', '2.18', 'Perl-5.20.0'', ('intel', '2015a')]

see 
http://easybuild.readthedocs.org/en/latest/Writing_easyconfig_files.html#dependencies



and possibly make this perl version a variable somewhere

Regards,
Jens Timmerman



— Many thanks for your reply,

Franky

Re: [easybuild] Rename a directory

[Jens Timmerman]

Hello Franky,

On 03/06/15 13:37, Backeljauw Franky wrote:

Hello,

We want to use EasyBuild to install a package which just needs a Tarball to be 
unpacked, but then we also want to change the directory name, how can we do 
that?



I believe you can set

|unpack_options = '-C %(installdir)s/'
|



Regards,
Jens Timmerman

— Regards,

Franky

Re: [easybuild] Two different EasyBlocks for the same package?

[Jens Timmerman]

Hi Ben,

On 09/01/15 00:19, Ben Roberts wrote:

Hi all,


Is there a convenient way to do things so I can have both EasyBlock modules in 
place at once? I don’t suppose I could call both EB_GROMACS, and it’s not clear 
that names such as EB_GROMACS_PRE46 and EB_GROMACS_POST46 will have the desired 
effect either. Thoughts?

You can use the self.version inside an easyblock,
so you could have one easyblock that behaves differently depending on 
the version in the easyconfig.

See SAMtools for an example [0]

One fun (nasty?) think you can do with python is multiple inheritance, 
so you can have the EB_GROMACS class inherit from both configuremake and 
cmakemake, or maybe inherit from EB_GROMACS_PRE46 and EB_GROMACS_POST46 
and in the init you could call the super depending on the version.


so something like this:

class EB_GROMACS(EB_GROMACS_PRE46, EB_GROMACS_POST46):
def __init__(self, *args, **kwargs):
if LooseVersion(self.version) < LooseVersion(4.6):
EB_GROMACS_PRE46.__init__(self, *args, *kwargs)
else:
EB_GROMACS_POST46.__init__(self, *args, *kwargs)


I'm sure there are other ways to do this, but this looks the most simple 
and modular at the same time to me.
If the internal changes are pretty small you can do it the way samtools 
does it and have version checks scattered trough the easyblock.

Thanks,
Ben


Regards,
Jens Timmerman

[0] 
https://github.com/hpcugent/easybuild-easyblocks/blob/0dfed6cbf56c4c21bc5397623cd91b606ff4/easybuild/easyblocks/s/samtools.py

Re: [easybuild] Install extensions to already installed package

[Jens Timmerman]

On 02/12/14 15:09, Ward Poelmans wrote:

On Tue, Dec 2, 2014 at 3:04 PM, Backeljauw Franky
 wrote:

Just a small question: is it possible to install extensions to a package 
instead of using a separate easyconfig-file?
E.g., I want to install some extra python modules to an already install Python 
installation where these modules did not appear in the extensions list when 
Python was installed using EasyBuild…

Just add them to the existing Python easyconfig and use the --skip
option. This will only install the newly added extensions.

Ward


You will probably also need the --force option.

Jens

Re: [easybuild] CMake-3.3.0-intel-2014b : problem building

[Jens Timmerman]

On 10/29/2014 04:50 PM, Backeljauw Franky wrote:


I understand — it’s an EB “issue” we have to live with. No problem.

Just to know: is that subject to change anywhere in the future (e.g., starting 
from EB version 2 or so)?
I believe this could be fixed if we were to make the cmake builds 
smarter, e.g. you could tell cmake to only look in certain 
subdirectories when looking for files, and ignore /usr/
However, I'm not a CMake specialist and further investigation is needed, 
but I did have a talk with the cmake developers who assured me this 
could be done.



Jens

Re: [easybuild] Python-3.4.1-intel-2014b.eb : freetype can not be built

[Jens Timmerman]

Dear Franky,

This is indeed a bug in the Python intel and foss -2014b easyconfig's,
I reported and fixed this last month, see
https://github.com/hpcugent/easybuild-easyconfigs/pull/1112

However this was not merged in yet.

Jens


On 10/28/2014 12:57 PM, Backeljauw Franky wrote:

Hello all,

When building Python-3.4.1-intel-2014b.eb, we got this error at the 
step for matplotlib:


== 2014-10-28 11:58:55,646 main.filetools INFO Creating directory
/apps/antwerpen/easybuild/build/Python/3.4.1/intel-2014a/matplotlib 
(parents:
True, set_gid: False, sticky: False)
== 2014-10-28 11:58:57,505 main.run ERROR EasyBuild crashed with
an error (at easybuild/tools/run.py:382 in parse_cmd_output): cmd
"python setup.py build " exited with exitcode 1 and output:

Edit setup.cfg to change the build options

BUILDING MATPLOTLIB
matplotlib: yes [1.3.1]
python: yes [3.4.1 (default, Oct 28 2014,
11:45:07)  [GCC
Intel(R) C++ gcc 4.4 mode]]
  platform: yes [linux]

REQUIRED DEPENDENCIES AND EXTENSIONS
 numpy: yes [version 1.8.1]
  dateutil: yes [using dateutil version 2.2]
   tornado: yes [tornado was not found. It is required
for the
WebAgg backend. pip/easy_install may
attempt to
install it after matplotlib.]
 pyparsing: yes [using pyparsing version 2.0.2]
 pycxx: yes [Official versions of PyCXX are not
compatible
with Python 3.x.  Using local copy]
libagg: yes [pkg-config information for 'libagg'
could not
be found. Using local copy.]
  freetype: no  [pkg-config information for
'freetype2' could
not be found.]
   png: yes [pkg-config information for 'libpng'
could not
be found. Using unknown version.]

OPTIONAL SUBPACKAGES
   sample_data: yes [installing]
  toolkits: yes [installing]
 tests: yes [using nose version 1.3.3]

OPTIONAL BACKEND EXTENSIONS
macosx: no  [Mac OS-X only]
qt4agg: no  [PyQt4 not found]
   gtk3agg: no  [gtk3agg backend does not work on
Python 3]
 gtk3cairo: no  [Requires cairo to be installed.]
gtkagg: no  [Requires pygtk]
 tkagg: no  [TKAgg requires Tkinter.]
 wxagg: no  [requires wxPython]
   gtk: no  [Requires pygtk]
   agg: yes [installing]
 cairo: no  [cairo not found]
 windowing: no  [Microsoft Windows only]

OPTIONAL LATEX DEPENDENCIES
dvipng: no
   ghostscript: no
 latex: no
   pdftops: no


* The following required packages can not
be built:
* freetype

== 2014-10-28 11:58:57,505 main.easyblock. WARNING build
failed (first 300 chars): EasyBuild crashed with an error (at
easybuild/tools/run.py:382 in parse_cmd_output): cmd "python
setup.py build " exited with exitcode 1 and output:

Edit setup.cfg to change the build options

BUILDING MATPLOTLIB

== 2014-10-28 11:58:57,506 main.EB_Python INFO Closing log for
application name Python version 3.4.1


It is complaining about freetype, but we have it installed on our 
system as an rpm. Likewise, it does not find libpng. Actually, we 
needed freetype-devel as well as libpng-devel to get the build do work...


However, mentioning both as osdependencies did not stop the build from 
starting, so it seems to just ignore the osdependencies. Is this as 
intended or what is wrong here?


That said: should we have freetype-devel and libpng-devel as rpm on 
the system, or should we have them as modules, preferably built with 
Easybuild and mentioned as dependencies (not osdependencies)?


-- Many thanks for your reply,

Franky

Re: AW: [easybuild] A patch containing a new file

[Jens Timmerman]

Olaf,

You might also like the RunCmdCP block I wrote, which will run a command
for each file in the sources list.
https://github.com/hpcugent/easybuild-easyblocks/pull/395

If this is not flexible enought, consider making it a bit more generic.
I'm sure we'll need more of this kind of tricks in the future anyway.

Regards
Jens Timmerman

On 04/28/2014 05:20 PM, Kenneth Hoste wrote:
>
> On 04/04/14 16:03, Olaf Walter wrote:
>> Hi Kenneth,
>>
>> thanks, option 1 an easy fix. I wouldn't consider it dirty, at least
>> to my standards...
>>
>> I put my extra Makefile (that calls the two Makefiles that come with
>> the software) next to the eb file. Then:
>>
>> patches = [('erds.Makefile', '.')] # copy Makefile to build directory
>> makeopts = '-f erds.Makefile' # and use this Makefile
>>
>> Option 2 look super elegant. In my case, it's the MakeCp block, not
>> ConfigureMake, so I tried
>> premakeopts = ["cd hmm && ", "cd phmm && "]
>>
>> As far as I can see, MakeCp will not iterate through the list. Only
>> hmm got was built.
>
> Hmm, weird, that should work...
>
> For each of the configure-make-make install cycles, EasyBuild will
> unpack the source tarball again, and start from scratch. Maybe that's
> what you're seeing?
>
>
>
> regards,
>
> Kenneth
>>
>> Thanks everybody for jumping in, and congratulations on the 1.12.0
>> release.
>>
>> Have a good weekend,
>> Olaf
>>
>> -Ursprüngliche Nachricht-
>> Von: easybuild-requ...@lists.ugent.be
>> [mailto:easybuild-requ...@lists.ugent.be] Im Auftrag von Kenneth Hoste
>> Gesendet: Freitag, 4. April 2014 15:04
>> An: easybuild@lists.ugent.be
>> Betreff: Re: [easybuild] A patch containing a new file
>>
>> Hi Olaf,
>>
>> On 04/04/14 14:45, Olaf Walter wrote:
>>> Dear easybuilders,
>>>
>>> the tool that I'm working on contains two source directories, each
>>> of them requiring a make. My idea was to create a patch that writes
>>> a top-level Makefile that calls both of them.
>> The tool source doesn't come with Makefiles provided?
>>
>>> This is what I get:
>>>
>>> EasyBuildError: "EasyBuild crashed with an error (at
>>> easybuild/1.11.1_p370_2/lib/python2.6/site-packages/easybuild_framework-1.11.1-py2.6.egg/easybuild/tools/filetools.py:610
>>> in apply_patch): Can't determine patch level for patch
>>> /gpfs01/home/bioadmn/eb20140311/eb/erds-1.1-Makefile.patch from
>>> directory
>>> /gpfs01/home/bioadmn/.local/easybuild/build/erds/1.1/goolf-1.4.12-no-OFED/erds1.1"
>>>
>>> This is my patch:
>>>
>>> diff -urN erds1.1.orig/Makefile erds1.1/Makefile
>>> --- erds1.1.orig/Makefile   1970-01-01 01:00:00.0 +0100
>>> +++ erds1.1/Makefile2014-04-04 14:10:49.015186000 +0200
>>> @@ -0,0 +1,9 @@
>>> +
>>> +all: make_hmm make_phmm
>>> +
>>> +make_hmm:
>>> +   cd hmm && $(MAKE)
>>> +
>>> +make_phmm:
>>> +   cd phmm && $(MAKE)
>>> +
>>> Does anybody have an idea how to solve this, or a different approach
>>> to my problem?
>>>
>>> Thanks,
>>> Olaf
>>
>> EasyBuild requires that the file to be patched is already there, so this
>> approach won't work.
>>
>> There's a couple of thing you can do:
>>
>> 1) EasyBuild support simply copying of files via the 'patches' entry:
>> ship the Makefile via patches, but not as a patch file:
>>
>> patches = [('Makefile', '.')]  # copy Makefile to build directory
>>
>> Something similar is done for OpenBLAS, see
>> https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.2.6-gompi-1.4.10-LAPACK-3.4.2.eb#L30
>>
>> .
>>
>> This is a rather 'dirty' approach though, since you need to add an extra
>> file.
>>
>>
>> 2) Use/abuse premakeopts and makeopts:
>>
>> prebuildopts = "cd hmm && "
>> buildopts = " && cd ../phmm && make"
>>
>> This will result in a build command line "cd hmm && make -j 8 && cd
>> ../phmm && make"
>>
>> (the "make -j 8" part is filled in by EasyBuild)
>>
>> This is already better, but note that you'd have to take care of "make
>> -j" yourself for phmm, and you have no idea what to set the value for
>> "-j" to (you don't know on which systems the easyconfig file would be
>> used).
>>
>>
>> 3) define prebuildopts as a *list* of strings; EasyBuild will cycle
>> through the configure/make/make install procedure multiple times
>>
>> prebuildopts = ["cd hmm && ", "cd phmm && "]
>>
>> This is also done for FFTW, where multiple calls to configure are
>> required to build the libraries with different precision, see
>> https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/f/FFTW/FFTW-3.3.3-gompi-1.4.10.eb#L16-21
>>
>> .
>>
>> I would definitely go for the last option.
>>
>>
>> regards,
>>
>> Kenneth
>>
>

Re: [easybuild] A patch containing a new file

[Jens Timmerman]

Dear Olaf

On 04/04/2014 02:45 PM, Olaf Walter wrote:
> Dear easybuilders,
>
> the tool that I'm working on contains two source directories, each of them 
> requiring a make. My idea was to create a patch that writes a top-level 
> Makefile that calls both of them. This is what I get:
>
> EasyBuildError: "EasyBuild crashed with an error (at 
> easybuild/1.11.1_p370_2/lib/python2.6/site-packages/easybuild_framework-1.11.1-py2.6.egg/easybuild/tools/filetools.py:610
>  in apply_patch): Can't determine patch level for patch 
> /gpfs01/home/bioadmn/eb20140311/eb/erds-1.1-Makefile.patch from directory 
> /gpfs01/home/bioadmn/.local/easybuild/build/erds/1.1/goolf-1.4.12-no-OFED/erds1.1"
>
> This is my patch:
>
> diff -urN erds1.1.orig/Makefile erds1.1/Makefile
> --- erds1.1.orig/Makefile   1970-01-01 01:00:00.0 +0100
> +++ erds1.1/Makefile2014-04-04 14:10:49.015186000 +0200
> @@ -0,0 +1,9 @@
> +
> +all: make_hmm make_phmm
> +
> +make_hmm:
> +   cd hmm && $(MAKE)
> +
> +make_phmm:
> +   cd phmm && $(MAKE)
> +
>
> Does anybody have an idea how to solve this, or a different approach to my 
> problem?
You can't just create a patch of a non existing file, it has to be
present before it can be patched.
You can add the new file in the sources, but it will not end up in the
unpacked directory of your tarball, so  you'll need an easyblock to fix
that.

You will probably want an EasyBlock anyway. So the aproach most people
will benefit from in the future is if you create an EasyBlock that
extends the ConfigureMake EasyBlock and
runs`ConfigureMake.build_step(self)` in each subdirectory it finds in
the unpacked tarball (e.g. using a glob pattern).

You could make it generic and call it ConfigureMakeSubdirs or something
along those lines.
So other packages that have the same behaviour could just be build by
specifying your EasyBlock.

If you're not familiar enough with python to try this on maybe someone
else could have a go at it...

Jens

Re: [easybuild] trouble with python modules and R packages built as extensions

[Jens Timmerman]

Hello Todd,

it seems like the python you are using to run EasyBuild has not been
compiled with
ssl support.

You can check this by checking if you can import the _ssl module
import _ssl

I just tested this on 3 different systems, and they seem to work.

I noticed pypi has a 301 permanently moved in place for all the http
requests, so https is mandatory.
I'm not sure how recently that has happened.

I also noticed we don't put OpenSSL as a dependency for the python
easyconfigs we ship,
so the pythons built with easybuild will have this behaviour if there is
no libssl and header files on the system.
maybe we should fix that?

Regards
Jens Timmerman

ps
http://stackoverflow.com/questions/2875402/does-urllib-or-urllib2-in-python-2-5-support-https

On 02/11/2014 06:37 PM, Heywood, Todd wrote:
> First, thanks for help on the earlier issue of module unloading, so that I
> can give you guys another issue :-).
>
> I am trying to build new versions of Python and R, and have long lists of
> required modules and packages. I¹m adding lines under the exts_list
> section of .eb files.
>
> I often get crashes due to "unknown url type² errors, which I think come
> from modules/packages trying to go find moudles/packages they depend on,
> and download them. I have been able to get by some by finding exact URL¹s
> and versions for dependencies and putting lines in the exts_list for these
> before the module/package in question. However, this is not working for
> one Python module I¹m trying to install now, and I figure I should ask
> about the real issue here instead of just trying to work around it.
>
> So, trying to install the matplotlib Python module with (eb ‹robot) with
> this:
>
> ('matplotlib', '1.3.1', {
> 'source_urls':
> ['https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotli
> b-1.3.1/'],
> }),
>
> The output, and log file excerpt are below. Matplotlib is apparently
> trying to download the ³distribute² dependency module. So I put this line
> above the matplotlib line:
>
> ('distribute', '0.6.28', {
> 'source_urls': ['https://pypi.python.org/packages/source/d/distribute/'],
> }),
>
>
> But the eaasybuild install still crashes with the same error.
>
> Output from build:
>
> == taking care of extensions...
> ERROR: EasyBuild encountered an exception (at
> easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_f
> ramework-1.10.0-py2.7.egg/easybuild/main.py:809 in
> build_and_install_software): autoBuild Failed (last 300 chars):
> D/lib/python2.7/urllib2.py", line 382, in _call_chain
> result = func(*args)
>   File
> "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p
> y", line 1247, in unknown_open
> raise URLError('unknown url type: %s' % type)
> urllib2.URLError: 
>
>
>
> From the log file:
>
> == 2014-02-11 11:35:44,089 main.filetools ERROR EasyBuild crashed with an
> error (at
> easybuild/software/EasyBuild/1.10.0/lib/python2.7/site-packages/easybuild_f
> ramework-1.10.0-py2.7.egg/easybuild/tools/fil\
> etools.py:799 in parse_cmd_output): cmd "python setup.py build" exited
> with exitcode 1 and output:
> Downloading
> http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.g
> z
> Traceback (most recent call last):
>   File "setup.py", line 10, in 
> use_setuptools()
>   File
> "/sonas-hs/it/hpc/home/eb/build/Python/2.7.5/goalf-1.5.12-no-OFED.10/matplo
> tlib/matplotlib-1.3.1/distribute_setup.py", line 163, in use_setuptools
> return _do_download(version, download_base, to_dir, download_delay)
>   File
> "/sonas-hs/it/hpc/home/eb/build/Python/2.7.5/goalf-1.5.12-no-OFED.10/matplo
> tlib/matplotlib-1.3.1/distribute_setup.py", line 132, in _do_download
> to_dir, download_delay)
>   File
> "/sonas-hs/it/hpc/home/eb/build/Python/2.7.5/goalf-1.5.12-no-OFED.10/matplo
> tlib/matplotlib-1.3.1/distribute_setup.py", line 212, in
> download_setuptools
> src = urlopen(url)
>   File
> "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p
> y", line 127, in urlopen
> return _opener.open(url, data, timeout)
>   File
> "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p
> y", line 410, in open
> response = meth(req, response)
>   File
> "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p
> y", line 523, in http_response
> 'http', request, response, code, msg, hdrs)
>   File
> "/opt/eb/software/Python/2.7.5-goalf-1.5.12-no-OFED/lib/python2.7/urllib2.p
> y", line 442, in error
> result = self._call_chain(*args)
>   File
> "/opt/

Re: [easybuild] Toolchain considerations and other newbie questions

[jens timmerman]

On 12/05/2013 10:43 PM, Olaf Walter wrote:
> Dear easybuilders,
> 
> I have made my first experiences with easybuild, and tried a few things. I am 
> really happy that I ran across your project, it is great work. But some 
> questions remain, maybe you can give me some advice.
> 
> (1) choosing a (single) toolchain
> We run a cluster for Bioinformatics software, with RHEL 6.4 and 10 GbE but no 
> Infiniband. If I understood correctly, goalf-1.1.0-no-OFED is the toolchain 
> of choice, right?

no, goalf toolchain has some flaws and we do not encourage anymore use
of it.

currently I use goolf-1.4.10 for all builds on my laptop, and since
yesterday we switched form ictce-4.1.13 to ictce-5.5.0 for intel
compiler builds.
So expect all new easyconfigs I add to be in these 2 toolchains.


> 
> (2) what if there is no eb for my toolchain?
> I noticed that for some programs, there are no goalf-1.1.0-no-OFED 
> easybuilds. Amoung these, there are essentials like Perl. 
> I understand that there will be conflicts if users try load a goalf-based 
> module (say, Perl) and a goolf-based module (say Python) at the same time 
> (zlib). Would you recommend to build Perl with goolf-1.4.10, or to create a 
> Perl-5.16.3-goalf-1.1.0-no-OFED.eb myself? Maybe there is a reason that there 
> is no such easyconfig? Should I trust the --try-options?
> 
staying within the gcc and intel toolchains I frequently do a
--try-toolchain=ictce,5.5.0  if this works out, consider opening a pull
request with the newly generated .eb file. Sometimes some manual
tweaking might be needed.

> (3) OS Independence 
> Besides the fact that easybuild bootstraps its own compiler, it still uses 
> some OS ressources. For example, if I use GCC/4.6.3 to compile helloworld.c 
> on a compute node (with no development tools installed), ld complains about 
> crt1.o not being there. So I am wondering: To what extend are the resulting 
> packages OS-independent? If I build on RHEL 6.4, will they run on RHEL 6.3, 
> 6.5, RHEL 5 (or even Ubuntu)?

I have not tested this, but they are still os specific, some things we
can not fix, like the infiniband kernel modules etc.
The reason EasyBuild was created was to be make it as easy to do a
recompile on a news system. Since we build everything architecture
specific chances are the binaries will not be able to run on the cpu of
your laptop anyway. Just rerun the eb command is what we encourage.

> 
> (4) Best practice for new releases
> How would you deploy new releases of software, for example when a new release 
> of easybuild comes out? I decided that a new release would go into a new 
> EASYBUILD_INSTALLPATH, and I inform users to update their .modulerc to 
> contain the new modulepath. Is this what you would recommend?
> 
We put everything in the same modulepath, so new and old versions of of
the same software are available by just doing a module load / module swap.

Since you can install EasyBuild with EasyBuild this is also what we do.



Regards
Jens Timmerman

Re: [easybuild] toolchain versions

[jens timmerman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 09/26/2013 11:40 PM, Bart Verleye wrote:
>
>
> A second question I have concerns github. Eb does not support github
sources, right? Is the solution then to download the source as a zip
file and make a link to the local zip?
>
Well, github supports arbitrary tar.gz downloading.

Try
https://github.com/hpcugent/easybuild-framework/archive/v1.7.0.tar.gz
or
https://codeload.github.com/hpcugent/easybuild-framework/tar.gz/v1.7.0

where v1.7.0 is a tag in git.
If you don't have any tags you can also use the commit id
e.g.
https://github.com/hpcugent/easybuild-framework/archive/a735f86b96ef4df0fec2bee39f723dc252ea887d.tar.gz
So in this way you can download the sources from easybuild without
having to create the tarballs yourself.

> Vriendelijke groet,
> Bart

Regards
Jens Timmerman
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.14 (GNU/Linux)

iEYEARECAAYFAlJFcuAACgkQrXYMyFNUlZZiNQCgq5W6gD87Pk2CgjTh5JP3/G/6
KH0AoJvoKCOPH3XXVzgZZj0sNm+M0t2K
=f6Ng
-END PGP SIGNATURE-

Re: [easybuild] application sources.

[jens timmerman]

On 08/06/2013 05:15 PM, Kenneth Hoste wrote:
>
> This does seem a right solution for Rgputools, just use the (short) commit ID 
> as a versionsuffix, and add a comment why we're doing it (because we know 
> particular versions change code over time).
>
> Do you need to use the long commit ID in the download URL, or does it also 
> work with the 5-6 character short-hand?
works with shorthands, as long as they are unique...
>
>>> Another way would be to put everything on github, and use this for our 
>>> mirroring, 
>>> we can fork existing projects that are already on github, and upload 
>>> sources from freely attributable software to some central repo.
>>> That is, we quickly need github support in easybuild then. :-p
>> I recall reading somewhere that github is not meant for above 1GB unless you 
>> ask permission etc.
>> The compilers already would exceed that or simply make it too clumsy.
>> For smaller bits though it might fly (but I still think of it, honestly, 
>> with a "hm")
> Also: pushing binary stuff in a git repo isn't exactly how it's intended to 
> be used.
>
>
> K.
No, the point is we can actually put the source code in, as github is
intended, to host source code, and let EB do a git checkout (or use the
archive function) , these tarballs are just an intermediary we don't
_really_ need.

But we do not need to parasite on github when we can set up a nice
mirror hosted by one or more of our institutions...
This was just posted as a possible alternative.

Regards
Jens Timmerman

Re: [easybuild] [ANN] EasyBuild v1.6.0

[jens timmerman]

On 07/17/2013 01:50 PM, Mohammed Gaafar wrote:
>
> We have EasyBuild-1.2.0 installed on our system. I need to upgrade to v1.6.0. 
> Will the bootstrap installation upgrade the current version to the newer one 
> if I set the installation path (--prefix) to the current installation 
> location? or, I have to make a fresh installation and then point the 
> environment variables to the new installation?
>
>
Mohammed,

I think the easiest way is to use the EasyBuild easyconfig file [0]

Just change the version to 1.6.0 and install it.
And then the new EasyBuild will be available with
module load EasyBuild/1.6.0
(make sure the right config files are still detected)

regards
Jens Timmerman

[0]
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/e/EasyBuild/EasyBuild-1.5.0.eb

Re: [easybuild] Re: Build gcc with up to date cloog and ppl

[jens timmerman]

Hi Ward,


On 07/15/2013 11:20 AM, Ward Poelmans wrote:
> OK, the problem did turn out was my sloppy git tree. Always clean up
> before you think you hit a bug :)
Glad you could get it to work :-)
> But I've got another question: when you specify a patch in a eb file
> and run the file, it doesn't look for the patch file in the directory
> of the eb file. It looks in the python tree. Is this intended
> behaviour? I think it would make sense to also look in the directory
> of the eb file. Should a write a patch for that? Or is this working as
> it is intended?
>
> Ward
You are right, this is how it currently works and is not intended
behavior but a known
limitation [0]
You are very welcome to fix this and send a pull request (or patch if
you prefer, but pull requests are easier for us)

I want to add that easybuild looks for the patch file in the source
directory and the robot path (iirc, if not, this also needs to be fixed)
So if you put your patch next to the downloaded tarball easybuild will
pick up on it.

Regards
Jens Timmerman

[0] https://github.com/hpcugent/easybuild-framework/issues/573

Re: [easybuild] Problem with CPATH

[jens timmerman]

On 07/03/2013 02:47 PM, Toon Verstraelen wrote:
> hi,
>
> I just made a easybuild config file for libint-2.0.3-stable.
> (attached) The module file mentions a CPATH, e.g. in my case:
>
> /home/toon/.local/easybuild/software/libint/2.0.3-stable-goolf-1.4.10/include
>
>
> while the include files are installed in a subdir:
>
> /home/toon/.local/easybuild/software/libint/2.0.3-stable-goolf-1.4.10/include/libint2
>
>
> This is the directory that should be in the CPATH variable,
>
> How should I fix the CPATH variable?
>
Hi Toon,

At this point this can only be fixed in an easyblock iirc.
You can look at the example on
https://github.com/hpcugent/easybuild/wiki/Development-guide
more specific, you'll need to add a make_module_req_guess function that
is very similar to the one in the example.


Regards
Jens Timmerman

Re: [easybuild] new user: problem to get started

[jens timmerman]

Hello Andreas,


On 06/27/2013 02:27 PM, Andreas Kuntzagk wrote:
> Hi,
>
> I wanted to test easybuild today. Unfortunately I cannot get it running.
> My systems are Ubuntu 10.04 (old, I know) and I installed using
> easy_install.
>
> I edited $HOME/.easybuild/config.cfg according to
> https://github.com/hpcugent/easybuild/wiki/Configuration
>
> """
> build_path="/home/kuntzagk/easybuild/build"
> install_path="/home/kuntzagk/easybuild"
> source_path="/home/kuntzagk/easybuild/source"
> repository = FileRepository("/home/kuntzagk/easybuild/repo")
> log_format = ("/home/kuntzagk/easybuild/log", "easybuild-%(name)s.log")
> log_dir ="/home/kuntzagk/easybuild/log"
> """
>
> Is this a problem with my Ubuntu or is it something with my config?
>
> regards, Andreas
>
This seems to be a problem with your config.

Currently we treat the config file as plain python code, and this
config's default location is shown in the help
as-C CONFIG, --config=CONFIG
Path to EasyBuild config file (def
/home/jens/.local/lib/python2.7/site-packages/easybuild_framework-1.5.0-py2.7.egg/easybuild/easybuild_config.py)

We want to get away from this format and use the configfile format as
used by pythons configparser. This support has already been added
to easybuild, and it show's up in the help as
--configfiles=CONFIGFILES
Parse (additional) configfiles (type
comma-separated list; def /home/jens/.easybuild/config.cfg)


The configuration as shown on the wiki still uses the old format, whilst
you have put this in the file where easybuild expects the new config
format.
I have tried to clarify this on the wiki.

To fix your probem I think it is enought to rename your config from
$HOME/.easybuild/config.cfg to $HOME/.easybuild/config.py.

Or you could switch to the new (undocumented, except for the messages in
eb -H) config format with config sections:
In this case your config file should look something like (untested)
"""
[config]
buildpath= /home/kuntzagk/easybuild/build
installpath= /home/kuntzagk/easybuild
sourcepath= /home/kuntzagk/easybuild/source
repository = FileRepository
repositorypath = /home/kuntzagk/easybuild/repo
logfile_format = /home/kuntzagk/easybuild/log,easybuild-%%(name)s.log
"""

TL;DR:
rename config.cfg to config.py

Regards
Jens Timmerman

[easybuild] ISC13

[Jens Timmerman]

Hi EasyBuilders,

Stijn De Weirdt (stdweirdt) and I (Jens Timmerman / jgtimmer) will be
present at
ISC13 in leipzig in a couple of weeks.

We will be presenting BoF 20:
*Best Practices in Building & Installing Scientific Software* [0]

We will be polling for best practices, explain our goals, and try to get
feedback on how we can improve EasyBuild, and the building of Scientific
Software in general.

If you will be attending ISC13, please give a shout,
maybe we can meet one evening.

Regards
Jens Timmerman


[0]
http://www.isc-events.com/isc13_ap/presentationdetails.php?t=contribution=2108=select=eventdetails

Re: [easybuild] Bioinformatics packages...

[Jens Timmerman]

Hi,

why are all these package to-do list?
They are already included
see
https://github.com/hpcugent/easybuild/wiki/List-of-supported-software-packages

abyss
https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/a/ABySS
bioperl:
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/p/Perl/Perl-5.16.3-goolf-1.4.10.eb
 and
https://github.com/hpcugent/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/b/BioPerl/BioPerl-1.6.1-goolf-1.4.10-Perl-5.16.3.eb
bowtie
https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs/b/Bowtie
bwa
https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/b/BWA

mrbayes
https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/m/MrBayes
beagle
https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/b/BEAGLE

regards
Jens Timmerman


On 23 May 2013 12:46, George Tsouloupas <g.tsoulou...@cyi.ac.cy> wrote:

> Dear EasyFriends,
>
> While duplication of work is a really good way to catch errors, we could
> reduce it a bit;
> Here are the packages we are working on these days at The Cyprus Institute
> with respect to Bioinformatics:
>
>
> "Packages" with a pull request already submitted or to be submitted within
> a few days (I'm not sure if we'll make the freeze):
> PLINK
> AllPathslg
> bam2fastq
> bwa
> EMBOSS
> paml
> GATK
>
> "Packages" with configs/blocks in easybuild.experimental
> MUSCLE
> CD-HIT
> RaxML
> RepeatMasker
> GPU-BLAST
> CUDA-BLASTP
>
> "Packages" on our immediate to-do list:
> BioPython
> Bioconductor
> Bioperl
> Oases
> bamtools
> blat
> bowtie
> fasta
> fastqc
> freebayes
> mrbayes
> beagle lib
> Picard-tools
> glimmer
>
> "Packages" on no-so-immediate to-do list:
> Phrap
> Phred
> Consed
> Abyss
> lastz
>
>
> All feedback is welcome!
>
> Thanks!
>
> George Tsouloupas,
> HPC Operations
> CaSToRC, the Cyprus Institute.
>
>
>
>

Re: EasyBuild v1.3.0rc2 (was: Re: [easybuild] call for testing: EasyBuild v1.3.0rc1)

[Jens Timmerman]

Hey,

The robot is now enabled by default now for some commands which would do
nothing without the option.
The only example I can thing of is --search

Regards
Jens Timmerman


On 2 April 2013 09:59, Jens Timmerman <jens.timmer...@gmail.com> wrote:

> The robot is now enabled by default now for some commands which would do
> nothing without the option.
> The only example I can thing of is --search thought.
>
>
> On 31 March 2013 22:06, Kenneth Hoste <kenneth.ho...@ugent.be> wrote:
>
>> Hi Fotis,
>>
>> On 31 Mar 2013, at 17:21, Kenneth Hoste wrote:
>>
>> >
>> >> What is more of a feature than a bug is, that the robot now kicks-in
>> automagically
>> >> (-r parameter); to be honest, I like that and, it saves my day...
>> let's keep it so :-)
>> >> (is it a default config parameter perhaps? haven't searched into it,
>> as of yet)
>> >
>> > Oh, but that *is* a bug.
>> >
>> > It's not a good idea to have it enabled by default, because that would
>> lead to surprises, e.g. when you have an incorrect MODULEPATH or
>> EASYBUILD_INSTALLPATH setting.
>> > You'd potentially end up rebuilding all dependencies with a default
>> enabled robot, which is not what you want most likely...
>> >
>> > We will need to fix this before we ship v1.3...
>>
>>
>> I was just checking this, and --robot is not enabled by default...
>>
>> An easy test:
>>
>> $ eb --version
>> This is EasyBuild 1.3.0rc2 (framework: 1.3.0rc2, easyblocks: 1.3.0rc2) on
>> host boegel.ugent.be.
>>
>> $ eb ictce-4.0.6.eb
>> == temporary log file in case of crash
>> /var/folders/6y/x4gmwgjn5qz63b7ftg4j_40mgn/T/easybuild-Qhv4BQ.log
>> == resolving dependencies ...
>> ERROR: EasyBuild crashed with an error (at easybuild/main.py:524 in
>> resolve_dependencies): Dependencies not met. Cannot resolve [('ifort',
>> '2011.6.233'), ('imkl', '10.3.6.233'), ('impi', '4.0.2.003'), ('icc',
>> '2011.6.233')]
>>
>>
>>
>> $ eb ictce-4.0.6.eb --robot
>> == temporary log file in case of crash
>> /var/folders/6y/x4gmwgjn5qz63b7ftg4j_40mgn/T/easybuild-JV4ksI.log
>> == resolving dependencies ...
>> == processing EasyBuild easyconfig
>> /Users/kehoste/.local/lib/python2.7/site-packages/easybuild_easyconfigs-1.3.0.0rc2-py2.7.egg/easybuild/easyconfigs/i/icc/icc-2011.6.233.eb
>> == building and installing icc-2011.6.233...
>> == fetching files...
>> ERROR: EasyBuild encountered an exception (at easybuild/main.py:810 in
>> build_and_install_software): autoBuild Failed (last 300 chars):
>> c2-py2.7.egg/easybuild/easyconfigs/l_ccompxe_intel64_2011.6.233.tgz,
>> /Users/kehoste/.local/easybuild/sources/i/icc/l_ccompxe_intel64_2011.6.233.tgz,
>> /Users/kehoste/.local/easybuild/sources/icc/l_ccompxe_intel64_2011.6.233.tgz,
>> /Users/kehoste/.local/easybuild/sources/l_ccompxe_intel64_2011.6.233.tgz
>>
>>
>> Why did you think the robot feature is enabled by default?
>>
>>
>> regards,
>>
>> Kenneth
>
>
>

Re: [easybuild] CUDA toolkit to be delivered, request for comments

[Jens Timmerman]

On Mon, 2013-01-14 at 11:23 +0100, Fotis Georgatos wrote:
> Good morning folks,

> 
> This multitude of packages is not all that nice, also because later on we 
> wish to add 
> CUDA as a dependency for GPU-enabled NAMD (ref. 
> ../../n/NAMD/NAMD-2.9-MVAPICH181-CUDA.eb)
> and we would rather codify the dep as 4.2.9 instead of "4.2.9ubuntu10.04.eb".
> 
> Do we have an elegant way to handle this? What would you suggest?
> 

is there actually any difference in the instal-linux.pl scripts between
these releases? 
I mean something that we do not fix already with sething the preload
path and include paths?

Otherwise I would op for just picking one, and making sure we can
install it on any/every platform...

Regards
Jens Timmerman

Re: [easybuild] Re: release candidate for EasyBuild version 1.0.0

[Jens Timmerman]

On Fri, 2012-11-09 at 18:46 +0100, Stijn De Weirdt wrote:
> >>> The framework was unaffected, the bugs were limited to particular
> >>> software packages.
> >>
> >> Nice. What is your recommended "authoritative" way to track such
> >> changes?
> >>
> >> I am having my view at this one now which is not always most
> >> readable summary:
> >> https://github.com/hpcugent/easybuild-easyconfigs/commits/master
> >> and I understand this addresses the zlib version fighting I
> >> noticed, too!
> >
> > I'd say checking the commit log is indeed the best way to go. Only a
> > limited number of commits are involved, and I tried to make the
> > commit messages clear.
> >
It might be easier to get a complete overview if the "merged pull
request" was just completely removed and replaced by the description.
> >
> >> Furthermore, some packages -including the bioinformatics ones we
> >> have contributed- may have a little bit difficulty to build
> >> straight out of the box. I suggest to play clean and lean for the
> >> v1.0 release and put them aside for a next release. (ie. if you see
> >> any suspicion of trouble, better put it in a queue for later). We
> >> do not need to be so strict for next releases, just this first
> >> one.
> >
> > They seem to be building fine on our end, although we have had some
> > issues with the MUMmer easyblock you contributed, see
> > https://github.com/hpcugent/easybuild-easyblocks/issues/8 .
> >
> > There will be several easyblocks that are not perfect in the current
> > release, but I don't see that as a reason to exclude them. We want to
> > get feedback, and if people think we're building something in the
> > wrong way, we want to get feedback on it, or even better, to let them
> > contribute fixes for our mistakes. So, I'd leave those in, especially
> > since the build works fine in a clean regtest.
> if you are aware of issues with configs or blocks that prevent 
> functional usage, we shouldn't ship them at all.
> 
> stijn

In this mummer case, the libraries get built and installed, but the
scripts not.
So it is valid as a dependency to get AMOS working, but it's not fully
functional... (And actually AMOS and MUMmer require perl, which is not
in EasyBuild yet)

Re: [easybuild] news from .lu side & pkg2eb, (was: EasyBuild v0.8)

[Jens Timmerman]

 no
> meaning on Debian
This is indeed a problem, we added detection for debian packages with
dpkg in the develop branch now. But since packages across distro's can
have different names this is no perfect solution.

A way to solve this would be by just adding a .eb file for tcsh and
treat it as a normal dependency, but we don't actually want to build an
entire linux distro this way. This also would not work for kernel
modules which are a real system dependency (e.g., for infiniband support
in certain mpi versions)

So feedback here is welcome.

> * Define the namespace "by a standard"; eg. like
> [https://github.com/fgeorgatos/HPC-RFC/blob/master/0001/0001.md]
> * Give freedom to define the namespace format (eg. original Package names OR
> the lower case version)
> * Support more configurable formats of version strings (as seen below, how
> other HPC sites do it)
>  * nested levels (ie. split version string in sub sections)
Agreed, we're going to add a configuration options for this.
> 
>  * provide hooks for gnu/gcc/intel/pgi etc strings
What exactly do you mean here?

> It may be that solutions can be provided outside of EasyBuild itself or an FAQ
> could give the obvious answers, if there are.
> 
I will make sure my explanations here get added to our wiki and/or a FAQ
section. And we're also in the process of completing them before the 1.0
release.

> One thing that is worthy to point out is the modules namespace & Easyblocks;
> The following, in effect prevents me from writing the fancy language "r":
> class_name = name[0].upper() + name[1:].replace('-','_')
> Yes, I mean "r" vs "R" ;-)
> I can accept the logic of needing to create a case-insensitive namespace
> for clarity on the user side but, is there another systems reason for it?

You could always create a class 'R_' and use easyblock="R_" in your .eb
file. You are correct that this will not work by default however.
The reason here is that we try to adhere to PEP8 where possible, which
declares [1]:
Almost without exception, class names use the CapWords [2] convention. 

If you believe this can be considered an exception we can leave it out
again.
> 
> If we can reply that last question, then the whole discussion can move on
> to the other SYSADMIN-oriented features, which in a way derive from that.
> 
> My stance is, that EasyBuild has all the foundations to become a standard
> tool for many more HPC sites but, it must give freedom to the sysadmin.

We greatly appreciate your feedback and are very happy this is proving a
useful tool for you. 

> 5) Epilogue
> 
> Finally, I am experimenting with some concepts on "freezing" github repos
> and creating running software for that; if someone else is doing something
> related, please keep in touch ;-)
What exactly do you mean by freezing?
> 
> cheers,
> Fotis
> 

Jens Timmerman

[0] https://github.com/mxcl/homebrew/tree/master/Library/Formula
[1] http://www.python.org/dev/peps/pep-0008/#class-names
[2] http://en.wikipedia.org/wiki/CamelCase

Re: [easybuild] Environment modules packaged for debian.

[Jens Timmerman]

On Tue, 2012-05-22 at 13:20 +0200, Fotis Georgatos wrote:
> fyi.
> for us the easy way out proved to be the manual way:
> * Download the .deb package from "testing" repo, then manually resolve deps.
> 
This brings us back on topic,
indeed the environment-modules package landed in debian-testing the day
after I announced it's landing in debian-unstable here. I changed the
links now and suspect this is the simplest way to get it indeed.

> This seems to do the job for now and passes the KISS criterion;
> we'll see how to automate it a bit in the future in some elegant way.
> 
> Fotis
> 

[easybuild] Environment modules packaged for debian.

[Jens Timmerman]

Good news everyone, 

The 'environment-modules' which easybuild depends on has been officially
packages for and included in the debian archives since 3rd of May 2012.
[0]
This also means it will get synced into the next ubuntu release (12.10
Quantal Quetzal [1]).

So now people will have an easy time getting dependencies for easybuild
in .deb based distro's (an rpm is already available for a long time)

I will link to the .deb file in the readme file and wiki. [2]


cheerz
Jens Timmerman


[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=583484
[1] https://wiki.ubuntu.com/QuantalQuetzal/ReleaseSchedule
[2] http://packages.debian.org/unstable/main/environment-modules

Re: [easybuild] Re: Subscription request to list easybuild

[Jens Timmerman]

On Fri, 2012-05-11 at 17:12 +0200, Kenneth Hoste wrote:
> Yo,
> 
> Wat doen we hier mee? Maken we een 2de lijst aan voor users 
> (easybuild-us...@lists.ugent.be), en houden we deze intern?
> 
> Andere opties?
> 
> 
> K.

Ik dacht dat easybuild@lists.ugent.be publiek zou zijn, stond zo in mijn
hepix presentatie.

Re: [easybuild] Fwd: previous MVAPICH2 versions

[Jens Timmerman]

http://mvapich.cse.ohio-state.edu/nightly/mvapich2/branches/1.7/

On Fri, 2012-05-11 at 15:57 +0200, Kenneth Hoste wrote:
> Vreemde manier van werken toch...
> 
>  Original Message  
>   Subject: 
> previous MVAPICH2 versions
>  Date: 
> Fri, 11 May 2012 09:53:15 -0400
> (EDT)
>  From: 
> Dhabaleswar Panda
> 
>To: 
> Kenneth Hoste
> 
>CC: 
> Dhabaleswar Panda
> 
> 
> 
> Hi,
> 
> Thanks for your note. Since we have 1.8 GA release for the last two weeks,
> we do not support 1.7 any longer and thus, it is not available.
> 
> Please use the latest 1.8 GA version.
> 
> Best Regards,
> 
> DK
> 
> =
> 
> Hello,
> 
> Is there a download URL for previous MVAPICH2 source releases?
> 
> I was trying to download MVAPICH2 v1.7 to test something, but since the
> 1.8 release it's nowhere to be found.
> 
> 
> regards,
> 
> Kenneth
> 
> 
> 

[easybuild] new repo

[Jens Timmerman]

Dear easybuild enthusiasts 

A new repo was created on savannah:
ssh://savannah.ugent.be/savannah/git/source/easybuild/easybuild1.git

This is a second repo under the easybuild group, so no new project on
savannah, user permissions are all the same.

enjoy.

Jens