Re: [easybuild] next EasyBuild conf call: Wed Mar 15th 2017, 5pm CET

[Jack Perdue]

Any chance:

toolchainsopts = {'rpath': True]

will be supported in 3.1.2 without requiring
--rpath --experimental  on the command line
as well?

I'm not a big RPATH fan (being raised on RH and Fedora),
but for packages Python-2.7.12-GCCcore-6.3.0-bare and
Python-3.5.2-GCCcore-6.3.0-bare which I use as a builddep
for a variety of Python modules without targeting a particular
final toolchain, its needed for derived modules sanity check
("Import foo") since libpython isn't in LD_LIBRARY_PATH since
it was just a builddep

Anyway. rpath toolchain opt would be cool for next release
if possible.  Seems to work as I need from the command line
(when I remember to add those to the command line)

jack








On 03/15/2017 12:43 PM, Alvarez, Damian wrote:

As always, notes in the wiki: 
https://github.com/hpcugent/easybuild/wiki/Conference-call-notes-20170315

The next telco will be on the 29th of March: 
https://plus.google.com/u/0/events/ctfejker527j88vq7sk7l8cmd0s?cfem=1

Cheers,
Damian

On 14/03/17 16:39, "easybuild-requ...@lists.ugent.be on behalf of Kenneth Hoste" 
<easybuild-requ...@lists.ugent.be on behalf of kenneth.ho...@ugent.be> wrote:

 Dear EasyBuilders,

 The next EasyBuild conf call is planned for Wednesday March 15th 2017,
 5pm CET;
 see also https://plus.google.com/events/c1f3gt69hoashfluir48htdsols .

 Current agenda:

* outlook to EasyBuild v3.1.2
* update on automatic style checking for easyconfig pull requests
* IRC vs Slack for `#easybuild` chat channel
* Q

 Suggestions for additional topics are welcome.
 Please let us know if you're planning to attend this conf call.

 More information about the EasyBuild conf calls is available at
 https://github.com/hpcugent/easybuild/wiki/Conference-calls .


 regards,

 Kenneth







Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

RE: [easybuild] HPC Testing Framework Collaboration

[Siddiqui, Shahzeb]

Hi Everyone

Thanks for all your comments, I think all of these ideas are great and I am 
hoping to get some exchange of ideas in order to find a way to generate test 
that require build from source.

I have released an update on my work. Please see 
https://github.com/shahzebsiddiqui/BuildTest

One thing I have nailed down is the ability to test binary commands. Simple 
test like gcc –version and any possible command can be done through a template 
script. I am trying to use a mixture of template scripts to autogenerate test 
scripts along with sed to modify module, version tag so that test can be module 
and version agnostic. As per Ken’s request, I have added check to see if module 
exist prior to generating a test. It is almost working with the exception of 
CUDA specific toolchains. Another feature I tried to implement is the use of 
CC, CXX, FC flags to properly find the compiler. Right now this is done through 
file extensions and a mixture of setup.sh script that can source the proper 
values. For Intel, CUDA, GCC the values for CC, FC, CXX would be different. 
This holds true for MPI wrappers. For MPI tests, I have a mpi template script 
that covers some generic mpi test.

There are some test that fall outside the scope of templates. For those I have 
a custom template for instance Python and Java.

It is difficult to create a universal template for each application because the 
compilation can vary as well as the execute step.

Please feel free to share more ideas, I hope this gains traction into something 
useful.

Regards,




From: xavier.besse...@gmail.com [mailto:xavier.besse...@gmail.com] On Behalf Of 
Xavier Besseron
Sent: Tuesday, March 14, 2017 6:57 PM
To: easybuild@lists.ugent.be; Siddiqui, Shahzeb
Subject: Re: [easybuild] HPC Testing Framework Collaboration

Hi Shahzeb,

This looks like an interesting contribution. I have a few comments for you 
below.

- I agree with the comments of Kenneth, especially when he says you need to 
"have some way of defining both version-agnostic and version-specific tests". I 
would extend the constraint to the module name as well. In the sense, you 
should find a way to have some tests applying to different modules. For 
example, for OpenMPI, MPICH, MVAPICH, most of the tests would be the same. I 
guess it applies to BLAS too.

So, I would say that for each test, you need a way to describe to which modules 
and versions it applies.
One idea for that could be use test labels (cf CTest documentation). For 
example:
add_test(NAME MPI-hello_c COMMAND sh hello_c.sh )
set_tests_properties(MPI-hello_c PROPERTIES LABELS mpi ) # this should work 
with any MPI

add_test(NAME OpenMPI-ompi_info COMMAND sh ompi_info.sh )
set_tests_properties(OpenMPI-ompi_info PROPERTIES LABELS openmpi ) # applies to 
OpenMPI, any version

add_test(NAME OpenMPI-2.0.1-featureXXX COMMAND sh test_feature_.sh )
set_tests_properties(OpenMPI-2.0.1-featureXXX PROPERTIES LABELS openmpi-2.0.1 ) 
# applies to OpenMPI version 2.0.1 only

Then, you can use option -L of CTest to select only a subset of tests. For 
example:

ctest -L '^(mpi|openmpi|openmpi-2.0.1)$' # to test OpenMPI/2.0.1
ctest -L '^(mpi|openmpi|openmpi-2.0.0)$' # to test OpenMPI/2.0.0
ctest -L '^(mpi|mpich|mpich-3.2)$'   # to test MPICH/3.2

This would also mean you need to reorganize the tests in your repository to 
replace the module name/version directory hierarchy by a kind of categorization.


- About the compilation of the tests:
I believe the compilation should be a test itself. Being able to run 
successfully mpicc hello.c -o hello_c is a good starting point to test an MPI.
Be aware that with CTest, you can also specify dependencies between tests and 
required files:
- DEPENDS: a test should run after another, eg make sure you run hello_c after 
compiling it
- REQUIRED_FILES: a test should run only if a file exists, eg make sure you run 
hello_c only if it has been created


- One thing that is missing so far is the possibility to run tests on two 
physical nodes. Most of the MPI errors/misconfigurations I got happened when 
trying to start 2 MPI processes on 2 different nodes. This probably means that 
you have to run the tests as a job submitted to the cluster and pass the list 
of nodes to run on.
This was actually my main motivation when I setup 
https://github.com/besserox/ATIS/tree/master/MPI
 and 
http://cdash.uni.lu/index.php?project=UL-HPC-Testing.


- If you have trouble installing