Hi Everyone Thanks for all your comments, I think all of these ideas are great and I am hoping to get some exchange of ideas in order to find a way to generate test that require build from source.
I have released an update on my work. Please see https://github.com/shahzebsiddiqui/BuildTest One thing I have nailed down is the ability to test binary commands. Simple test like gcc –version and any possible command can be done through a template script. I am trying to use a mixture of template scripts to autogenerate test scripts along with sed to modify module, version tag so that test can be module and version agnostic. As per Ken’s request, I have added check to see if module exist prior to generating a test. It is almost working with the exception of CUDA specific toolchains. Another feature I tried to implement is the use of CC, CXX, FC flags to properly find the compiler. Right now this is done through file extensions and a mixture of setup.sh script that can source the proper values. For Intel, CUDA, GCC the values for CC, FC, CXX would be different. This holds true for MPI wrappers. For MPI tests, I have a mpi template script that covers some generic mpi test. There are some test that fall outside the scope of templates. For those I have a custom template for instance Python and Java. It is difficult to create a universal template for each application because the compilation can vary as well as the execute step. Please feel free to share more ideas, I hope this gains traction into something useful. Regards, From: xavier.besse...@gmail.com [mailto:xavier.besse...@gmail.com] On Behalf Of Xavier Besseron Sent: Tuesday, March 14, 2017 6:57 PM To: easybuild@lists.ugent.be; Siddiqui, Shahzeb Subject: Re: [easybuild] HPC Testing Framework Collaboration Hi Shahzeb, This looks like an interesting contribution. I have a few comments for you below. - I agree with the comments of Kenneth, especially when he says you need to "have some way of defining both version-agnostic and version-specific tests". I would extend the constraint to the module name as well. In the sense, you should find a way to have some tests applying to different modules. For example, for OpenMPI, MPICH, MVAPICH, most of the tests would be the same. I guess it applies to BLAS too. So, I would say that for each test, you need a way to describe to which modules and versions it applies. One idea for that could be use test labels (cf CTest documentation). For example: add_test(NAME MPI-hello_c COMMAND sh hello_c.sh ) set_tests_properties(MPI-hello_c PROPERTIES LABELS mpi ) # this should work with any MPI add_test(NAME OpenMPI-ompi_info COMMAND sh ompi_info.sh ) set_tests_properties(OpenMPI-ompi_info PROPERTIES LABELS openmpi ) # applies to OpenMPI, any version add_test(NAME OpenMPI-2.0.1-featureXXX COMMAND sh test_feature_XXXX.sh ) set_tests_properties(OpenMPI-2.0.1-featureXXX PROPERTIES LABELS openmpi-2.0.1 ) # applies to OpenMPI version 2.0.1 only Then, you can use option -L of CTest to select only a subset of tests. For example: ctest -L '^(mpi|openmpi|openmpi-2.0.1)$' # to test OpenMPI/2.0.1 ctest -L '^(mpi|openmpi|openmpi-2.0.0)$' # to test OpenMPI/2.0.0 ctest -L '^(mpi|mpich|mpich-3.2)$' # to test MPICH/3.2 This would also mean you need to reorganize the tests in your repository to replace the module name/version directory hierarchy by a kind of categorization. - About the compilation of the tests: I believe the compilation should be a test itself. Being able to run successfully mpicc hello.c -o hello_c is a good starting point to test an MPI. Be aware that with CTest, you can also specify dependencies between tests and required files: - DEPENDS: a test should run after another, eg make sure you run hello_c after compiling it - REQUIRED_FILES: a test should run only if a file exists, eg make sure you run hello_c only if it has been created - One thing that is missing so far is the possibility to run tests on two physical nodes. Most of the MPI errors/misconfigurations I got happened when trying to start 2 MPI processes on 2 different nodes. This probably means that you have to run the tests as a job submitted to the cluster and pass the list of nodes to run on. This was actually my main motivation when I setup https://github.com/besserox/ATIS/tree/master/MPI<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_besserox_ATIS_tree_master_MPI&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=Pu-0eH9lAgBYbu0Gi_r-ke4qTUsVrVYbfpxX7AGRQ3A&e=> and http://cdash.uni.lu/index.php?project=UL-HPC-Testing<https://urldefense.proofpoint.com/v2/url?u=http-3A__cdash.uni.lu_index.php-3Fproject-3DUL-2DHPC-2DTesting&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=AlWMacIP8NfUtRiecsNTxLLc2sct6Cbpbbt7lv8PxI4&e=>. - If you have trouble installing CDash, maybe you can get started with http://my.cdash.org/<https://urldefense.proofpoint.com/v2/url?u=http-3A__my.cdash.org_&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=lgPXO3fNjO-mDoOO3pPozvd2kgY7zgjw37Ui1u1Q8dQ&e=> to get an overview of it (free for 10 submissions per day). Then it's very easy to switch to your self-hosted instance once you got it working. Regards, Xavier On Thu, Mar 9, 2017 at 2:58 PM, Siddiqui, Shahzeb <shahzeb.siddi...@pfizer.com<mailto:shahzeb.siddi...@pfizer.com>> wrote: Hi Fotis, I was trying to get CDash working but I had a pain getting all the dependencies working. PHP version on RHEL7 was not satisfying minimum req. I am looking to see if docker can work in resolving this. To my understanding CDash would be configured by the user and it needs apache and mysql or postgres. This would be a pain getting all of these setup which is outside the scope of this project. Hopefully docker would alleviate this pain. Regards, -----Original Message----- From: easybuild-requ...@lists.ugent.be<mailto:easybuild-requ...@lists.ugent.be> [mailto:easybuild-requ...@lists.ugent.be<mailto:easybuild-requ...@lists.ugent.be>] On Behalf Of Fotis Georgatos Sent: Thursday, March 9, 2017 12:06 AM To: easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be> Subject: Re: [easybuild] HPC Testing Framework Collaboration Hi Shanzeb, just 2 comments: * I'd love to see the ctest/cdash type of testing take off; here's an example from our past endeavours: http://cdash.uni.lu/index.php?project=UL-HPC-Testing<https://urldefense.proofpoint.com/v2/url?u=http-3A__cdash.uni.lu_index.php-3Fproject-3DUL-2DHPC-2DTesting&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=AlWMacIP8NfUtRiecsNTxLLc2sct6Cbpbbt7lv8PxI4&e=> * this one caught my attention recently; it might work in favour of literate testing (and, indirectly, useful documentation): https://bitbucket.org/michilu/shell-doctest<https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_michilu_shell-2Ddoctest&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=T2pQumwxWESZjjATwVPH2ftQSl-IsXJHc-cjgIIHIbE&e=> HPL against popular toolchain variants would make a good first focus of such effort, wouldn't it?! enjoy, F. On Mar 7, 2017, at 4:31 PM, Siddiqui, Shahzeb <shahzeb.siddi...@pfizer.com<mailto:shahzeb.siddi...@pfizer.com>> wrote: > Hello, > > I have been working on creating a testing framework to automatically generate > test cases for applications that work with EasyBuild in a HMNS module layout. > I couldn't seem to find any tool that provides test cases for HPC apps that > work properly with a module environment so I decided to create me own. > > I have pushed some test cases for intel, gcc, openmpi, mpich as > examples. If anyone is interested in developing a testing framework, check > out > https://github.com/shahzebsiddiqui/testgen-HPC<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_shahzebsiddiqui_testgen-2DHPC&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=_q4FI7qRdE06QL9aM-D_OwU0-XBfh8IEMYAGuvjdi8g&e=> > and maybe we can collaborate on a standard for basic test cases for each > application. > > Regards, > > > > > Shahzeb Siddiqui > HPC Linux Engineer > B2220-447.2 > Groton, CT cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum -- Dr Xavier BESSERON Research associate FSTC, University of Luxembourg Campus Kirchberg, Office E-007 Phone: +352 46 66 44 5418 http://luxdem.uni.lu/<https://urldefense.proofpoint.com/v2/url?u=http-3A__luxdem.uni.lu_&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=OQx2kMGTUsPmn3M3iRuS5Fo1UPuE6XBw9YQ55YI4jfI&s=1xlslhAVg65leAaoHnIISqv68HdIYCIJC4UPD-aRULo&e=>
