Hi Patrick,
> Looking at the software you described it seems a wonderful tool for humans
> to create, execute test steps and analyse test results entered manually by
> a human.
Actually, this was the primary goal - we wanted to support testing of systems
that are close to hardware (such as coreboot). Especially with coreboot,
personally, I found too often that I bought a board that was officially
"supported" just to find out that some things were actually broken while they
seemed to have been working in past versions well (happened to me lately with a
Thinkpad T410, see my previous postings to the list about this). The idea in
SystemTestPortal is to support testing for such regressions - but this of
course requires the availability of humans that execute these tests.
> What we are looking for is only the "data store" and visual representation
> of such, where automated tests are run by robots. Those (self-hosted or
> propietary) robots need to publish their test results somewhere using a web
> API.
I see. Well, there is a different project named "ReportPortal.io" that (imho)
does exactly that. We had a joint stand at FOSDEM 2019 together with them and
KiwiTCMS. It might be worth looking into that.
> Does SystemTestPortal support input from robots over for example REST API?
Not at the moment but it is a feature request we received multiple times so we
will eventually add this in the future.
> Does it support the idea of different products/configurations?
Yes, it does. We have a two-dimensional concept of a products and variants, but
I think coreboot would need three dimensions or even more, right? So for
example you could have:
- coreboot 4.14 ("clean" without patches)
- on a Thinkpad T410
- built with config options X and Y enabled but Z disabled
In addition, there could be also different configurations of the target
machine, different OSs on which you would test etc. - the key challenge here is
to decide what to put into the test cases themselves and what to put in the
products/variants/configuration metadata. Maybe you could try to describe a
data model that would be ideal from the coreboot perspective?
Cheers, Daniel
_______________________________________________
coreboot mailing list -- [email protected]
To unsubscribe send an email to [email protected]
