Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?
All in all it'd be much nicer if there was some kind of hook in ctest which could be used to inform it that a number of tests had been run, and let it populate and submit the XML report based on that information. But, if there's any way to do that, it's highly undocumented... It doesn't appear like there is a class for storing the test results before serializing the XML document to a file. Sounds like that's what would have to be written. cmCTestTestHandler::GenerateDartOutput Looks like it may be a place to start? Alternatively, how about making the custom test return a simple XML which had name pass/fail < full test text output> /> name pass/fail /> ... /etc/etc and then add a function into cmCtestHandler to parse the result - in the fixed form you define - back into a standard Dart compatible form which it then submits as usual. A slight modification of ADD_TEST to trigger the new form of output would probably be enough. JB ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?
On Tue, Jun 8, 2010 at 4:32 AM, Chris Hillery wrote: > On Tue, Jun 8, 2010 at 1:00 AM, Biddiscombe, John A. wrote: > >> I'm sure someone has a better answer ... but ... >> >> > I have complete control over the test >> > program and can make it output whatever I want. >> >> Have a look at the XML produced by ctest itself and generate it yourself? >> then submit it and you should see each test separately >> >> (Just thinking out loud) >> > > Yeah, worst case I'll head down that path. It's a hairy path though. For > instance, I'd still definitely want to run "normal" ctest tests as well, > which means the XML would need to be merged in some fashion. Also, it seems > like doing it that way opens you up to possible problems later if ctest or > cdash change the XML format at all. > > All in all it'd be much nicer if there was some kind of hook in ctest which > could be used to inform it that a number of tests had been run, and let it > populate and submit the XML report based on that information. But, if > there's any way to do that, it's highly undocumented... > It doesn't appear like there is a class for storing the test results before serializing the XML document to a file. Sounds like that's what would have to be written. cmCTestTestHandler::GenerateDartOutput Looks like it may be a place to start? -- Philip Lowman ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?
On Tue, Jun 8, 2010 at 1:00 AM, Biddiscombe, John A. wrote: > I'm sure someone has a better answer ... but ... > > > I have complete control over the test > > program and can make it output whatever I want. > > Have a look at the XML produced by ctest itself and generate it yourself? > then submit it and you should see each test separately > > (Just thinking out loud) > Yeah, worst case I'll head down that path. It's a hairy path though. For instance, I'd still definitely want to run "normal" ctest tests as well, which means the XML would need to be merged in some fashion. Also, it seems like doing it that way opens you up to possible problems later if ctest or cdash change the XML format at all. All in all it'd be much nicer if there was some kind of hook in ctest which could be used to inform it that a number of tests had been run, and let it populate and submit the XML report based on that information. But, if there's any way to do that, it's highly undocumented... Ceej aka Chris Hillery ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?
I'm sure someone has a better answer ... but ... > I have complete control over the test > program and can make it output whatever I want. Have a look at the XML produced by ctest itself and generate it yourself? then submit it and you should see each test separately (Just thinking out loud) JB -- John Biddiscombe, email:biddisco @ cscs.ch http://www.cscs.ch/ CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82 ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
[CMake] CTest: Any way around the "one test == one program invocation" assumption?
I asked this question in a followup on the CDash mailing list the other day, but it's really not strictly a CDash question so I'll toss it here too. There are a number of situations where I would like a single test program to perform a number of tests (in one case, nearly 20,000). I'm wondering if there's any way to make CTest see this as more than one test. I'm particularly interested in having the results uploaded to CDash in a useful fashion - individual test success/failure, maybe even separate test output and timing information. It would also be nice if running ctest locally at least displayed the list of failed tests. (I'm not expecting things like "ctest -R " to work right.) I have complete control over the test program and can make it output whatever I want. Has anyone ever tried this? Failing that, how difficult would it be to generate CDash-compatible XML output from my testdriver? And how would I go about uploading those results? Ceej aka Chris Hillery ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake