On 09/15/2015 11:03 AM, intrigeri wrote: > Alan wrote (23 Aug 2015 13:28:41 GMT) : >> but I encourage the people working on the test suite to have a look >> at OpenQA and consider wether it would make sense to use it. It has >> a nice web interface to update screenshots,
Sorry for not responding earlier! I looked at OpenQA ~4 years ago, before we started our own thing, but it seemed to me that it solved quite little of what we actually needed, at least on the top; it didn't support fine-grained hardware configuration (and not hotplugging at all), for instance. I felt weary of using something that we'd have to extend endlessly to make it useful for us, and while it would have been a potentially better longer-term solution, I'm not sure we'd have started to benefit from it even now. (Also, perl!) When bertagaz made the initial version of the cucumber + sikuli + libvirt based thing we have now, I was blown away at how easily extensible it was (despite me having no previous experience with Ruby). I still think that is true, despite some painful surprises (JRuby). We need so much flexibility to do all the things we want to do, so I'm not sure if anything else than a solution of our own would work efficiently. Or maybe I'm just having a case of NIH syndrome. :) >> and we would share maintenance with other distros. This I find really sad, however. > I've had a look during DebConf, and indeed the web interface for > developers is much better than what Jenkins will give us as-is. > Perhaps at some point we'll need $something that extracts data from > Jenkins and presents it to developers, and at that point it may be > worth looking into OpenQA. I'm unsure what the "$something that extracts data from Jenkins and presents it to developers" part refers to. If it refers to Alan's "it has a nice web interface to update screenshots", then I don't see the benefit in it. It's quite rare that only pictures need updating when stuff changes, and when they do, often more than one (used in the same scenario) needs updating. Our --retry-find option will let us solve that in (ideally) one babysitted run, while relying on automation + a web gui to update images could result in some crazy long back and forth, or that you must run a VM in parallel to take screenshots from. I'm not convinced we'll have much use of that feature except for trivial cases where only one image needs updating. And in trivial cases we can just extract the new image from the test suite failure screenshot. > Note that tests are written directly in Perl, with no overlay language > like Gherkin. Not as if we were taking much benefit from Gherkin for > {intra,inter}-team communication yet, but I like it that we're able to; > and I intend to try to use Gherkin more in the future when discussing > changes with UX folks. I personally like the clarity of what is (well, *should be*) going on that they provide to me, as a developer, even without the "easier communication with non-developers" part. Cheers! _______________________________________________ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.