On 11 Jul 01, Paul Larson wrote:
> On Thu, Jun 30, 2011 at 11:24 PM, Daniel Lezcano
> <daniel.lezc...@linaro.org>wrote:
>
> > When all tests will be finished I wish to switch the new way test suite
> > execution in lava, if it is possible.
> >
> Given that the old tests are broken at the moment and disabled, any reason
> we shouldn't switchover now?
I think we should switchover now. There is no real dependency on the old
tests.
> > Will the following format be ok ?
> >
> > test_01/cpu0 : checking scaling_available_frequencies file ... PASS
> > ...
>
> That would probably translate internally to something like:
> {test_id="test_01_cpu0", message="checking scaling_available_frequencies
> file", result="PASS"}
> Is that ok? Seems like something we should have no trouble making sense out
> of later I think. Also, the exact output is saved as an attachment as well.
Does LAVA send email in case of a FAIL result?
> > The result for a test case is PASS or FAIL.
> >
> > We support "unknown" as a result as well, if that helps you at all.
> Sometimes results can be indeterminate, or unimportant (odd as that may
> sound, sometimes the message is really what you're after, as illustrated by
> some of the previous pm qa tests)
>
> But under some circumstances, we need to do some extra work where a
> > failure does not mean the test case failed but the pre-requisite for the
> > test case is not met.
> >
> ...and we also support "skip" as a result. That seems like a correct use
> for it.
> You don't have to report it literally as "skip", you can call it "oink" for
> all we care, and just provide a translation table that converts whatever
> your result strings are to {pass, fail, skip, unknown}. For example, see
> the test definition for ltp.
>
>
> > deviation 0 % for 2333000 is ... VERY
> > >> GOOD
> > >
> > > Same comments as above about having an easier to interpret format, but
> > the
> > > result here: "VERY GOOD" - what does that mean? What are the other
> > possible
> > > values? Is this simply another way of saying "PASS"? Or should it
> > actually
> > > be a measurement reported here?
> >
> > Yep, I agree it is an informational message and should go to a logging
> > file. I will stick to a simple result 'PASS' or 'FAIL' and I will let
> > the user to read the documentation of the test in the wiki page to
> > understand the meaning of these messages (GOOD, VERY GOOD...).
> >
> keeping to the template you mentioned earlier, I wonder if we could do
> something like this:
> deviation_0_for_2333000: VERY GOOD ... PASS
> (are the numbers there consistent and useful as a test case name? I'm
> assuming so here)
> That would allow you to capture "VERY GOOD" as details in the message (one
> of the fields we keep). Also, your test could be smart enough to know that
> good, or verygood = pass, while bad, verybad = fail. Possibly I'm making a
> lot of assumptions here, but I think you see what I mean.
Yes, it makes sense to make GOOD, VERY GOOD, etc. informational (as part of
the messages) and map them to PASS.
> Keeping to a consistent results format in your output is a good practice,
> and makes this *much* easier for capturing the data important to you. Of
> course, anything is possible. We have some tests with rather elaborate
> results and for those, the test definition just inherits from a base class
> and defines it's own parser. If you're not feeling very pythonic, you could
> provide your own parser as part of the test download written in shell, c,
> ruby, go, whatever, that just acts as a filter, then have it just read it
> all in directly. There are lots of options, but I'm of the opinion that a
> consistent format makes it easier on humans looking at it as much as machine
> parsers. And since you have control over that, easiest to do it now. :)
>
> Thanks,
> Paul Larson
_______________________________________________
linaro-dev mailing list
linaro-dev@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-dev
