On Tuesday 2016-09-27 19:36 +0100, James Graham wrote: > On 27/09/16 19:20, L. David Baron wrote: > > >I'd like to see us run the reftests with our existing reftest > >harness, which is optimized to do image comparison in compiled code, > >and which also has a bunch of useful failure-annotation features [1] > >and features that test for other sorts of failures such as > >NS_ASSERTIONs. > > I think there are a number of ways that this is non-trivial; for example I'm > not aware that we have the infrastructure needed to generate reftest > manifests from the source tree, I don't think the existing reftest harness > can handle the possibility of a test causing a crash cleanly,
I don't think this is important. Crashes are rare enough that, in the rare case that we do have a non-intermittent crash, we can simply skip the test until it's fixed. > I don't think > we have the ability to autogenerate expectation data to account for the fact > that many tests will fail, and so on. I would rather start with wptrunner, > which already solves these problems, and improve the harness with better > performance and error reporting since these can be added incrementally > whereas the other things are prerequisites for any frequently updated > testsuite from upstream. I don't think it's acceptable to convert any existing reftest coverage in wptrunner's current state, or to ask our engineers to write tests for new features targeting wptrunner rather than the existing reftest harness. The existing reftest harness has had a significant amount of investment in the ability for engineers to debug local failures and get useful information out of failures in continuous integration. Since we haven't really had a lot of that experience with wptrunner, and the experience we have had has been ignored rather than issues being fixed (see, say, https://bugzil.la/1265575 and https://bugzil.la/1265582 , which are completely unacceptable), and as a result of such basic things being unfixed and the general lack of use, I don't think you have even remotely close to the full list of issues that need to be fixed in order to bring wptrunner up to parity. I think you'd be better off trying to solve the autogeneration and manifest issues in the existing harness. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914)
signature.asc
Description: PGP signature
_______________________________________________ dev-tech-layout mailing list [email protected] https://lists.mozilla.org/listinfo/dev-tech-layout
