On Tue, Aug 15, 2017 at 5:45 AM, Tim Flink wrote:
> On Wed, 9 Aug 2017 14:31:49 +0200
> Lukas Brabec wrote:
>
> > Hey, gang!
> >
> > As I read through standard interface and tried ansiblized branch of
> > libtaskotron, I found things that were not exactly clear to me and I
> > have some questions. My summer afternoon schedule involves feeding
> > rabbits (true story!) and I keep missing people on IRC, hence this
> > email.
> >
> >
> > = Test output and its format =
> >
> > Standard test interface specifies that [1]:
> > 1) "test system must examine the exit code of the playbook. A zero
> > exit code is successful test result, non-zero is failure"
> > and
> > 2) "test suite must treat the file test.log in the artifacts folder as
> > the main readable output of the test"
> >
> > ad 1) Examining the exit code is pretty straight forward. The mapping
> > to outcome would be zero to PASSED and non-zero to FAILED. Currently
> > we use more than these two outcomes, i.e. INFO and NEEDS_INSPECTION.
> > Are we still going to use them, if so, what would be the cases? The
> > playbook can fail by itself (e.g. fail like command not found, or
> > permission denied), but I presume this failure would be reported to
> > ExecDB not to ResultsDB. Any thoughts on this?
>
>
> I think that, for now at least, we won't be using the INFO and
> NEEDS_INSPECTION. The standard interface definition is what it is and
> they've decided that results are binary. It's possible that could
> change in the future but for now, the INFO and NEEDS_INSPECTION states
> probably won't be used when reporting to resultsdb.
>
When it comes to distgit tests, yes. Our generic tasks can still use the
full range of supported outcomes, we just need to define a nice way to
export them (e.g. looking for ./artifacts/taskotron-results.yml).
However, Lukas has a very good point about failure which are not actual
test failures (the opposite of passing), but crashes. In Taskotron we
distinguish those - crashes don't get reported to ResultsDB, but stay in
ExecDB (before that, we used to report them as CRASHED into ResultsDB).
That allows maintainer to distinguish between these two states, and allows
us to potentially re-run the task for crashes (but not for failures). The
standard interface (SI) doesn't distinguish those, which I find quite a
step backwards. Any network hiccup or random error will now force package
maintainers to read the logs and understand what went wrong, instead of us
being able to re-run the test. This might be a good topic for discussion
with the CI team on Flock, perhaps.
>
> > ad 2) The standard interface does not specify the format of test
> > output, just that the test.log must be readable. Does this mean that
> > the output can be in any arbitrary format and the parsing of it would
> > be left to people who care, i.e. packagers? Wouldn't be this a problem
> > with if, for example, bodhi wanted to extract/parse this information
> > from ResultsDB and show it on update page?
>
Bodhi will not deal with test.log at all. Bodhi will look into ResultsDB
for PASSED/FAILED information and display that, and that's it.
>
> As far as I know, this was left vague on purpose so that it can change
> and be specified by the the test system. In this case, we'd definitely
> need to support the xunit xml format that I suspect most folks will be
> using but I'm open to the idea of keeping our results yaml format alive
> if it makes sense.
>
> So long as we're explicit about which formats are supported for our
> implementation of a test system and don't make silly choices, I think
> we're good here.
>
I'm a bit confused here. I suspected we would support our internal
ResultYAML format for our generic tasks, available e.g. at
./artifacts/taskotron-results.yml, on top of SI. It might be handy for
task-rpmlint (specifying more outcomes, notes, etc) and necessary for
multi-item tasks (task-rpmdeplint, task-upgradepth). Are you saying we
could support even more formats here, and if we find e.g.
./arfifacts/xunit-results.xml, we process it and use it for submitting more
detailed results into ResultsDB?
>
> > = Triggering generic tasks =
> >
> > Standard interface is centered around dist-git style tasks and doesn't
> > cover generic tasks like rpmlint or rpmdeplint. As these tasks are
> > Fedora QA specific, are we going to create custom extension to
> > standard interface, used only by our team, to be able to run generic
> > tasks?
>
> It'd be nice if it wasn't just us using it but the standard interface
> may indeed require some tweaking to get it to cover all of the usecases
> that we're interested in.
>
> Do you have any suggestions as to how we could make non-dist-git tasks
> work reasonably well without making drastic changes to what the
> standard interface currently is?
>
I believe the biggest obstacle here is reporting. The SI assumes this is
all that is needed:
* test subject
* exit code
For reporting to ResultsDB, we will reuse test subject
