Hi Stuart,
On 05/03/15 17:55, Stuart Barkley wrote:
On Wed, 4 Mar 2015 at 16:06 -0000, Kenneth Hoste wrote:
There are some ways (e.g., doing the build in a jail-like
environment), but they're not trivial.
My initial hope was that easybuild supported building software outside
of the live environment (mock, chroot, fakeroot). This is not the
case and I can live with it for now but I hope this is/becomes part of
the future direction.
We want to head into this direction, Actually it has been a whishlist
item since before version 1.0 came out.
So if anybody is interested in working
on getting this feature in I would love to point him/her into the right
direction
to get started on working on this, maybe a simple chroot is all we need,
maybe it's a lot more complicated, I haven't had time to look into this yet.
But I wouldn't mind helping people who want to look into implementing this.
Just to clarify: the easyconfigs that are part of the EasyBuild
installation are meant to be treated as examples (which does not
mean they should contain inconsistencies like the one you reported).
This is also where I find the current easybuild process a little
awkward. I generally prefer examples to be out of the main execution
path. i.e. If these are really examples, they should not be used
automatically.
They are not, you need to provide -r before they will be picked up.
and people have actually complained about having to provide -r every time.
If they are a default configuration, they should work
and be maintained (which is a lot of work for 2000+ configurations).
We actually build every single easyconfig before every release, yes,
this is a lot of work, and we are aware that it's not because it's working
on our system it will work everywhere. And you are completely free
(and able to) deviate from them.
I presume this can be addressed by not installing the easyconfig
package, but I would still like to have the examples handy for
reference.
They are also there so you don't need to replicate all the work
someone else has already done.
I think Kenenths main point was that you should not be afraid
to make a change in an easyconfig if it beter suits your local needs.
This is different for EasyBlocks, if you need to make a change
there, we really want to know about it, because probably everybody
using EasyBuild will benefit from it, whereas changes in easyconfigs
will in general only apply to a small subset of people.
I did see that numpy and scipy and a few others were already included
in the Python build. I didn't notice the version numbers differences
but it is good to see support for other versions available. Can these
modules be loaded on top of a python already containing older
versions?
sometimes they can, sometimes they can't.
If the python version is the same they should work,
but numpy and scipy sometimes break between python versions because
they use the internal ABI, which sometimes changes between python versions.
So if you try to load these with a different python loaded you will get
a conflict.
However if your python matches the python in the numpy and scipy
modules, then yes,
they can be loaded, since the module will prepend their path to the
python path, so
python will pick up these.
What is current thinking on whether things like scipy should be built
into the python build or be separate modules?
Generally packages that require another module as a dependency get put
into a separate module. e.g lxml depends on libxml2 and libxslt
So putting this in the main python build would require more dependencies
for this python module, dependencies that maybe not everybody needs.
If it's just pure python packages they can go into the python build.
For now I'm planning to build a single consistent set of software
(with just the goolf toolchain). Every 6-8 months or so would be an
updated release with freshened software.
You can look into the foss-year[a|b] toolchain then.
We have decided to use a new toolchain twice a year,
For us this will be the intel-year[a|b] toolchain, but a foss toolchain
will
also always follow soon after.
So we will mainly add new easyconfigs using these toolchians,
except where a piece of software really needs the newest version of the
compiler.
These changes just allow building to complete successfully. I
have not started testing any applications beyond the build
process.
Some builds include tests, like the numpy easyblock that checks the
performance of numpy.dot in an attempt to ensure that numpy was
correctly linked with BLAS (see
https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/n/numpy.py#L197).
And that's beside the unit test suite which is also run.
Lately, we've been paying more attention to including (simple) tests
in the build procedure as executed by EasyBuild, but we should do
this more, where possible.
This is good and I do notice test cases being run in the log files.
I'm still getting used to the log files and am finding some issues in
their usability, but that is subject for a separate discussing at
another time.
some tips about understanding the logfiles are provider here:
http://easybuild.readthedocs.org/en/latest/Logfiles.html
Other things I plan to do are building documentation, demonstration,
example and performance cases for each major software package. In
part, this becomes site specific but I also hope for some support from
the HPCBIOS and similar efforts.
We'd love to hear more about this!
regards,
Jens Timmerman
Stuart
