Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?

2010-06-08 Thread Biddiscombe, John A.


All in all it'd be much nicer if there was some kind of hook in ctest which 
could be used to inform it that a number of tests had been run, and let it 
populate and submit the XML report based on that information. But, if there's 
any way to do that, it's highly undocumented...



It doesn't appear like there is a class for storing the test results before 
serializing the XML document to a file.  Sounds like that's what would have to 
be written.





cmCTestTestHandler::GenerateDartOutput

Looks like it may be a place to start?





Alternatively, how about making the custom test return a simple XML which had



  name

  pass/fail

  < full test text output>

/>



  name

  pass/fail

  

/>

... /etc/etc

and then add a function into cmCtestHandler to parse the result - in the fixed 
form you define - back into a standard Dart compatible form which it then 
submits as usual. A slight modification of ADD_TEST to trigger the new form of 
output would probably be enough.



JB


___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?

2010-06-08 Thread Philip Lowman
On Tue, Jun 8, 2010 at 4:32 AM, Chris Hillery wrote:

> On Tue, Jun 8, 2010 at 1:00 AM, Biddiscombe, John A. wrote:
>
>> I'm sure someone has a better answer ... but ...
>>
>> > I have complete control over the test
>> > program and can make it output whatever I want.
>>
>> Have a look at the XML produced by ctest itself and generate it yourself?
>> then submit it and you should see each test separately
>>
>> (Just thinking out loud)
>>
>
> Yeah, worst case I'll head down that path. It's a hairy path though. For
> instance, I'd still definitely want to run "normal" ctest tests as well,
> which means the XML would need to be merged in some fashion. Also, it seems
> like doing it that way opens you up to possible problems later if ctest or
> cdash change the XML format at all.
>
> All in all it'd be much nicer if there was some kind of hook in ctest which
> could be used to inform it that a number of tests had been run, and let it
> populate and submit the XML report based on that information. But, if
> there's any way to do that, it's highly undocumented...
>

It doesn't appear like there is a class for storing the test results before
serializing the XML document to a file.  Sounds like that's what would have
to be written.

cmCTestTestHandler::GenerateDartOutput
Looks like it may be a place to start?

-- 
Philip Lowman
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?

2010-06-08 Thread Chris Hillery
On Tue, Jun 8, 2010 at 1:00 AM, Biddiscombe, John A. wrote:

> I'm sure someone has a better answer ... but ...
>
> > I have complete control over the test
> > program and can make it output whatever I want.
>
> Have a look at the XML produced by ctest itself and generate it yourself?
> then submit it and you should see each test separately
>
> (Just thinking out loud)
>

Yeah, worst case I'll head down that path. It's a hairy path though. For
instance, I'd still definitely want to run "normal" ctest tests as well,
which means the XML would need to be merged in some fashion. Also, it seems
like doing it that way opens you up to possible problems later if ctest or
cdash change the XML format at all.

All in all it'd be much nicer if there was some kind of hook in ctest which
could be used to inform it that a number of tests had been run, and let it
populate and submit the XML report based on that information. But, if
there's any way to do that, it's highly undocumented...

Ceej
aka Chris Hillery
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest: Any way around the "one test == one program invocation" assumption?

2010-06-08 Thread Biddiscombe, John A.
I'm sure someone has a better answer ... but ...

> I have complete control over the test
> program and can make it output whatever I want.

Have a look at the XML produced by ctest itself and generate it yourself? then 
submit it and you should see each test separately

(Just thinking out loud)

JB


-- 
John Biddiscombe,    email:biddisco @ cscs.ch
http://www.cscs.ch/
CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland  | Fax:  +41 (91) 610.82.82



___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake