Hi Ken, Thanks for your feedback. I will definitely try to incorporate some of these changes. I am not a fan of CMake either, but it seems like a good option since it can work with CDash and that would be a great way to monitor all the test. One of the troubling task I have is to find an elegant method to build test that require compilation with least amount of parameters.
I have decided to store source code in "src/" directory and hopefully build out a collection of code that can work. This would be useful when you need a particular test for instance "saxpy" can be placed in "src/CUDA" and then I can use the test for any version of CUDA. Ø Also try to avoid copy-pasting tests (cfr. OpenMPI), but maybe have some way of defining both version-agnostic and version-specific tests I am also trying to add capability to export an entire test suite from one application to another. This requires some string manipulation for the test. I think I would have to change the template in order to avoid duplicating code. Regards, From: [email protected] [mailto:[email protected]] On Behalf Of Kenneth Hoste Sent: Thursday, March 9, 2017 4:20 AM To: [email protected] Subject: Re: [easybuild] HPC Testing Framework Collaboration Hi Shahzeb, On 07/03/2017 17:31, Siddiqui, Shahzeb wrote: Hello, I have been working on creating a testing framework to automatically generate test cases for applications that work with EasyBuild in a HMNS module layout. I couldn't seem to find any tool that provides test cases for HPC apps that work properly with a module environment so I decided to create me own. I have pushed some test cases for intel, gcc, openmpi, mpich as examples. If anyone is interested in developing a testing framework, check out https://github.com/shahzebsiddiqui/testgen-HPC<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_shahzebsiddiqui_testgen-2DHPC&d=DwMD-g&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=J4Jg2KIT7s6-hYvtu6Kf6ziOp346jI1MIqJxLZb7qPk&s=ADQi4tdYkDN4PFa0MyhrpCCmS7DHdn_UB9dbJVEfDb4&e=> and maybe we can collaborate on a standard for basic test cases for each application. I haven't looked into this in detail, but it sure looks interesting. To be honest, I'm not a big fan of CMake myself, but I'm not familiar with ctest so I'll give it the benefit of doubt here. ;-) A major thing missing right now is a proper open source license. This could be a major help: https://choosealicense.com/<https://urldefense.proofpoint.com/v2/url?u=https-3A__choosealicense.com_&d=DwMD-g&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=J4Jg2KIT7s6-hYvtu6Kf6ziOp346jI1MIqJxLZb7qPk&s=Wb2dDynkLbpEQkfSF0qYGdCyI-qefiTKiqXa_VV37Sw&e=>, although in your case you probably also have company approval to worry about? One other thing I think you should do early on is to define some utility functions in bash to remove some of the heavy copy-pasting. For example, you should have 'load_module' function where you load a specified module, and check the exit code, and maybe also check that the module indeed appears in 'module list'. You should also have a function like 'run_and_check' that runs a specified command, captures stdout/stderr, and checks the exit code. Also try to avoid copy-pasting tests (cfr. OpenMPI), but maybe have some way of defining both version-agnostic and version-specific tests. These are just some random thoughts, I like the approach overall. Maybe this could grow out to be a much used testing framework.... regards, Kenneth Regards, Shahzeb Siddiqui HPC Linux Engineer B2220-447.2 Groton, CT
