Hi Shahzeb, This looks like an interesting contribution. I have a few comments for you below.
- I agree with the comments of Kenneth, especially when he says you need to "have some way of defining both version-agnostic and version-specific tests". I would extend the constraint to the module name as well. In the sense, you should find a way to have some tests applying to different modules. For example, for OpenMPI, MPICH, MVAPICH, most of the tests would be the same. I guess it applies to BLAS too. So, I would say that for each test, you need a way to describe to which modules and versions it applies. One idea for that could be use test labels (cf CTest documentation). For example: add_test(NAME MPI-hello_c COMMAND sh hello_c.sh ) set_tests_properties(MPI-hello_c PROPERTIES *LABELS mpi* ) # this should work with any MPI add_test(NAME OpenMPI-ompi_info COMMAND sh ompi_info.sh ) set_tests_properties(OpenMPI-ompi_info PROPERTIES *LABELS openmpi* ) # applies to OpenMPI, any version add_test(NAME OpenMPI-2.0.1-featureXXX COMMAND sh test_feature_XXXX.sh ) set_tests_properties(OpenMPI-2.0.1-featureXXX PROPERTIES *LABELS * *openmpi-2.0.1* ) # applies to OpenMPI version 2.0.1 only Then, you can use option -L of CTest to select only a subset of tests. For example: ctest -L '^(*mpi*|*openmpi*|*openmpi-2.0.1*)$' # to test *OpenMPI/2.0.1* ctest -L '^(*mpi*|*openmpi*|*openmpi-2.0.0*)$' # to test *OpenMPI/2.0.0* ctest -L '^(*mpi*|*mpich*|*mpich-3.2*)$' # to test *MPICH/3.2* This would also mean you need to reorganize the tests in your repository to replace the module name/version directory hierarchy by a kind of categorization. - About the compilation of the tests: I believe the compilation should be a test itself. Being able to run successfully *mpicc hello.c -o hello_c *is a good starting point to test an MPI. Be aware that with CTest, you can also specify dependencies between tests and required files: - DEPENDS: a test should run after another, eg make sure you run hello_c after compiling it - REQUIRED_FILES: a test should run only if a file exists, eg make sure you run hello_c only if it has been created - One thing that is missing so far is the possibility to run tests on two physical nodes. Most of the MPI errors/misconfigurations I got happened when trying to start 2 MPI processes on 2 different nodes. This probably means that you have to run the tests as a job submitted to the cluster and pass the list of nodes to run on. This was actually my main motivation when I setup https://github.com/besserox/ATIS/tree/master/MPI and http://cdash.uni.lu/index.php?project=UL-HPC-Testing. - If you have trouble installing CDash, maybe you can get started with http://my.cdash.org/ to get an overview of it (free for 10 submissions per day). Then it's very easy to switch to your self-hosted instance once you got it working. Regards, Xavier On Thu, Mar 9, 2017 at 2:58 PM, Siddiqui, Shahzeb < shahzeb.siddi...@pfizer.com> wrote: > Hi Fotis, > > I was trying to get CDash working but I had a pain getting all the > dependencies working. PHP version on RHEL7 was not satisfying minimum req. > I am looking to see if docker can work in resolving this. To my > understanding CDash would be configured by the user and it needs apache and > mysql or postgres. This would be a pain getting all of these setup which is > outside the scope of this project. Hopefully docker would alleviate this > pain. > > Regards, > > -----Original Message----- > From: easybuild-requ...@lists.ugent.be [mailto:easybuild-request@ > lists.ugent.be] On Behalf Of Fotis Georgatos > Sent: Thursday, March 9, 2017 12:06 AM > To: easybuild@lists.ugent.be > Subject: Re: [easybuild] HPC Testing Framework Collaboration > > Hi Shanzeb, > > just 2 comments: > * I'd love to see the ctest/cdash type of testing take off; > here's an example from our past endeavours: > http://cdash.uni.lu/index.php?project=UL-HPC-Testing > * this one caught my attention recently; it might work in favour of > literate testing (and, indirectly, useful documentation): > https://bitbucket.org/michilu/shell-doctest > > HPL against popular toolchain variants would make a good first focus of > such effort, wouldn't it?! > > enjoy, > F. > > On Mar 7, 2017, at 4:31 PM, Siddiqui, Shahzeb <shahzeb.siddi...@pfizer.com> > wrote: > > > Hello, > > > > I have been working on creating a testing framework to automatically > generate test cases for applications that work with EasyBuild in a HMNS > module layout. I couldn't seem to find any tool that provides test cases > for HPC apps that work properly with a module environment so I decided to > create me own. > > > > I have pushed some test cases for intel, gcc, openmpi, mpich as > > examples. If anyone is interested in developing a testing framework, > check out https://github.com/shahzebsiddiqui/testgen-HPC and maybe we > can collaborate on a standard for basic test cases for each application. > > > > Regards, > > > > > > > > > > Shahzeb Siddiqui > > HPC Linux Engineer > > B2220-447.2 > > Groton, CT > > cheers, > Fotis > > > -- > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # signal detected in a CERN forum > > > > > > > > -- Dr Xavier BESSERON Research associate FSTC, University of Luxembourg Campus Kirchberg, Office E-007 Phone: +352 46 66 44 5418 http://luxdem.uni.lu/
