> Hi folks!
>
> I am currently rolling out some changes to the Fedora openQA deployment
> which enable a new testing workflow. From now on, a subset of openQA
> tests should be run automatically on every critpath update, both on
> initial submission and on any edit of the update.
>
> For the next little while, at least, this won't be incredibly visible.
> openQA sends out fedmsgs for all tests, so you can sign up for FMN
> notifications to learn about these results. They'll also be
> discoverable from the openQA web UI - https://openqa.fedoraproject.org
> . The results are also being forwarded to ResultsDB, so they'll be
> visible via ResultsDB API queries and the ResultsDB web UI. But for
> now, that's it...I think.
>
> Our intent is to set up the necessary bits so that these results will
> show up in the Bodhi web UI alongside the results for relevant
> Taskotron tests. There's an outside possibility that Bodhi is actually
> already set up to find these results in ResultsDB, in which case
> they'll just suddenly start showing up in Bodhi - we should know about
> that soon enough. :) But most likely Bodhi will need a bit of a tweak
> to find them.
Let me add a bit of a technical background here. Bodhi web UI now queries
ResultsDB for all available testcases, and then asks for
item=$item&type=koji_build for all these testcases. So the new results won't be
visible (unless you start reporting them per build).
Our depcheck and upgradepath tasks report both type=koji_build (for each SRPM)
and type=bodhi_update (for each bodhi update). Both these tools internally
process RPMs or builds, and then query Bodhi at the end and collate the results
to bodhi_update results. We want to get rid of that collating, because it has
numerous issues:
1. It's slow, because Bodhi is very slow to respond
2. It often causes the task to fail, because Bodhi often returns 500 errors
3. It's prone to race conditions. It happens often that a Bodhi update is
edited between the task start and end, changing included builds.
4. It's unnecessary, because Bodhi knows all this information, and can collate
the data itself, without any network issues or race conditions.
So while we still publish type=bodhi_update for compatibility reasons (I'm not
sure whether some part of bodhi backend still might use this data), but want to
get rid of it.
So my first thought was to recommend you to also publish just type=koji_build
results and finish this transition. But then I realized that's wrong. OpenQA
operates completely different than the aforementioned tasks do. We operate on
builds, and can distinguish which build is causing issues. Collating them into
bodhi_update results is just a convenience measure for the consumer. But you
operate on the whole update as a set. You can't distinguish which build of the
update caused the issues, you just know that some of them did. So the smallest
unit for you is bodhi update, and you should report results as such.
The way forward is, I believe, to extend Bodhi to query both type=koji_build
for all included builds and collate the results (if needed), and also query
type=bodhi_update and shows those results as well. Because different tasks
operate on different type of data, which influences how they publish the
results. And both use cases are valid.
There's one important thing we need to do first, though. Bodhi ID doesn't
identify the thing tested uniquely, because Bodhi updates are mutable (and the
ID is kept). So Bodhi (or any gating tools) can't rely on just retrieving the
latest result for a particular Bodhi ID and trust that result. It might be old
and no longer reflect the current state. We need to extend bodhi_update results
with "timestamp" key in extra data, that will report the "last_modified" time
of the Bodhi update tested. And Bodhi (or any other tool) must not only query
for item=$bodhi_id&type=bodhi_update, but also for ×tamp=$timestamp. Only
with this we can be sure we've really tested particular Bodhi update.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
