Re: [RFC] [tests] Running the wayland tests against compositor-headless
On Mon, Apr 22, 2013 at 11:02 PM, Eoff, Ullysses A ullysses.a.e...@intel.com wrote: -Original Message- From: Sam Spilsbury [mailto:smspil...@gmail.com] snip snip Yes, it would be trivial to enable the input-based tests to run on the headless backend... however, it's unclear to me whether the headless backend should be the one responsible for initializing/tracking the keyboard and pointer inputs. That kinda feels counter intuitive to the definition of headless (but perhaps not :-/). Maybe we could have the weston-test extension module initialize the inputs in an idle loop callback, for example if the output-model == headless, then do: weston_seat_init_pointer(seat); weston_seat_init_keyboard(seat, NULL); They'll probably need to be lazy-initialized at least. At the moment weston is segfaulting because the tests trigger callbacks in weston that expect the pointer and keyboard interfaces to be present on a seat. Either that or a headless seat implementation might be in order. For the other two, which ones are they and what kind of protocol support is lacking? Perhaps there is another way we can get them to work on headless? The broader reason I'm looking into this is because I want to see if we can eventually export the weston-test module as part of the installation as well as the headless and no-op backends so that applications can run integration tests against weston to verify that they handle input, shell surface changes etc correctly. I'm looking into doing some GSoC work on a project where I suspect that the wayland backend will not be run all the time, and it will be good to have integration tests in place to ensure that it doesn't break accidentally because developers may not be able to run it regularly. The headless backend module gets installed, already. But if you're also talking about exporting headless backend header(s), then I'm willing to bet that won't happen. As for the weston-test module being exported (module and headers), that would have to be the maintainer's decision (Kristian; aka, krh) and/or a consensus from other community contributors. If nothing else, you can always write your own test extension, too, that will enable you to do integration testing on your particular application use-cases. Lets have a discussion about this when Kristian gets back. I think having an upstreamed module providing wl_test will be beneficial for lots of downstream application developers because it means they can use some common protocol which they know works and integrate that into their applications. In the event that we don't want to do that, another alternative would be to look into making weston-test an external module as part of a broader external modules proposal so it can have its own direction in terms of what it supports. Of course the final alternative is to just have applications ship their own modules and test protocols, but this is not really ideal. So - shall I create and submit some patches to get the tests working on headless? I suppose this is a desirable thing right? I wouldn't be the one to make the ultimate decision on their inclusions into upstream. It'd be nice to hear what others think about this topic, too. Either way, it wouldn't hurt to send patches for review, feedback, and discussion... even if they end up getting rejected for a different solution ;). I'll write up some patches ASAP. Best, Sam snip Cheers, U. Artie Thanks, Sam -- Sam Spilsbury ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel
RE: [RFC] [tests] Running the wayland tests against compositor-headless
-Original Message- From: wayland-devel-bounces+ullysses.a.eoff=intel@lists.freedesktop.org [mailto:wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org] On Behalf Of Eoff, Ullysses A Sent: Monday, April 22, 2013 1:24 PM To: Sam Spilsbury; wayland Subject: RE: [RFC] [tests] Running the wayland tests against compositor-headless Changing the tests to run headless is probably not necessary... To clarify... I only meant this in the context of the weston-tests-env script , i.e. it does not need to be changed in order for you to automate 'make check' to run them on headless ;) ...but if it's a problem that the tests themselves can't run on the headless backend, then changing those in a way to work on headless, too, might be useful. You can create your own custom check script if weston-test-env does not suite your needs. To use it with make check, just set/export the LOG_COMPILER environment variable to refer to that script before running make check. make check just invokes your LOG_COMPILER script with the current test as an argument and then uses your scripts exitcode to determine pass/fail status. See http://www.gnu.org/software/automake/manual/html_node/Parallel- Test-Harness.html I use a custom python script in this manner so that I can run the tests on both Weston drm and x11 backends per test with only one invocation of make check. My custom script also allows me to parse the test output/results into a standard, normalized Xml format that is recognized by our continuous integration server. U. Artie Eoff -Original Message- From: wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org [mailto:wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org] On Behalf Of Sam Spilsbury Sent: Monday, April 22, 2013 11:50 AM To: wayland Subject: [RFC] [tests] Running the wayland tests against compositor-headless Hi, I'm not sure if this is a work-item that's already in-progress by somebody. I noticed that weston has a headless compositor backend and a no-op renderer, which effectively do not depend on having OpenGL or *a* windowing system available at the time that weston is run. It would be really great if we could run the tests against this backend, because then it means that they can be run on foreign machines like continuous integration servers with ease. I've tried quickly changing the backend over in weston-test-env - currently most of the tests are failing because client-pointer is not initialized. Upon initializing that, most of them pass except those that depend on client-keyboard . I suspect that this will involve having to create a wl_keyboard with a set of dummy xkb data for layouts, though I'm not sure how to do that yet. My question to the list is - does it make sense to submit a patch to change the tests over to use the headless backend? Or is the preference to keep them using the wayland or x11 backend for now. Best, Sam. -- Sam Spilsbury ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel
Re: [RFC] [tests] Running the wayland tests against compositor-headless
On Mon, Apr 22, 2013 at 3:48 PM, Eoff, Ullysses A ullysses.a.e...@intel.com wrote: -Original Message- From: wayland-devel-bounces+ullysses.a.eoff=intel@lists.freedesktop.org [mailto:wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org] On Behalf Of Eoff, Ullysses A Sent: Monday, April 22, 2013 1:24 PM To: Sam Spilsbury; wayland Subject: RE: [RFC] [tests] Running the wayland tests against compositor-headless Changing the tests to run headless is probably not necessary... To clarify... I only meant this in the context of the weston-tests-env script , i.e. it does not need to be changed in order for you to automate 'make check' to run them on headless ;) Ah interesting. I guess the point of the tests is to verify the backends themselves as opposed to testing the core compositor (well, it tests that as an incident, but I guess its not the point?). Does it make sense to have also acceptance tests which really only verify the core behavior of weston and not the backends themselves? ...but if it's a problem that the tests themselves can't run on the headless backend, then changing those in a way to work on headless, too, might be useful. So at the moment - they aren't - well, its possible to run the two library based ones. 3 of the client based ones are easy to enable, as for the other three, two of them protocol support from the backend, the other one (xwayland) doesn't really make sense on headless. The broader reason I'm looking into this is because I want to see if we can eventually export the weston-test module as part of the installation as well as the headless and no-op backends so that applications can run integration tests against weston to verify that they handle input, shell surface changes etc correctly. I'm looking into doing some GSoC work on a project where I suspect that the wayland backend will not be run all the time, and it will be good to have integration tests in place to ensure that it doesn't break accidentally because developers may not be able to run it regularly. So - shall I create and submit some patches to get the tests working on headless? I suppose this is a desirable thing right? Best, Sam You can create your own custom check script if weston-test-env does not suite your needs. To use it with make check, just set/export the LOG_COMPILER environment variable to refer to that script before running make check. make check just invokes your LOG_COMPILER script with the current test as an argument and then uses your scripts exitcode to determine pass/fail status. See http://www.gnu.org/software/automake/manual/html_node/Parallel- Test-Harness.html I use a custom python script in this manner so that I can run the tests on both Weston drm and x11 backends per test with only one invocation of make check. My custom script also allows me to parse the test output/results into a standard, normalized Xml format that is recognized by our continuous integration server. U. Artie Eoff -Original Message- From: wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org [mailto:wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org] On Behalf Of Sam Spilsbury Sent: Monday, April 22, 2013 11:50 AM To: wayland Subject: [RFC] [tests] Running the wayland tests against compositor-headless Hi, I'm not sure if this is a work-item that's already in-progress by somebody. I noticed that weston has a headless compositor backend and a no-op renderer, which effectively do not depend on having OpenGL or *a* windowing system available at the time that weston is run. It would be really great if we could run the tests against this backend, because then it means that they can be run on foreign machines like continuous integration servers with ease. I've tried quickly changing the backend over in weston-test-env - currently most of the tests are failing because client-pointer is not initialized. Upon initializing that, most of them pass except those that depend on client-keyboard . I suspect that this will involve having to create a wl_keyboard with a set of dummy xkb data for layouts, though I'm not sure how to do that yet. My question to the list is - does it make sense to submit a patch to change the tests over to use the headless backend? Or is the preference to keep them using the wayland or x11 backend for now. Best, Sam. -- Sam Spilsbury ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel -- Sam Spilsbury ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org
Re: [RFC] [tests] Running the wayland tests against compositor-headless
So - shall I create and submit some patches to get the tests working on headless? I suppose this is a desirable thing right? Ah, to clarify this point - I meant submitting patches which would allow the headless backend to be used in testing, as opposed to just x11 and drm. Best, Sam You can create your own custom check script if weston-test-env does not suite your needs. To use it with make check, just set/export the LOG_COMPILER environment variable to refer to that script before running make check. make check just invokes your LOG_COMPILER script with the current test as an argument and then uses your scripts exitcode to determine pass/fail status. See http://www.gnu.org/software/automake/manual/html_node/Parallel- Test-Harness.html I use a custom python script in this manner so that I can run the tests on both Weston drm and x11 backends per test with only one invocation of make check. My custom script also allows me to parse the test output/results into a standard, normalized Xml format that is recognized by our continuous integration server. U. Artie Eoff -Original Message- From: wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org [mailto:wayland-devel- bounces+ullysses.a.eoff=intel@lists.freedesktop.org] On Behalf Of Sam Spilsbury Sent: Monday, April 22, 2013 11:50 AM To: wayland Subject: [RFC] [tests] Running the wayland tests against compositor-headless Hi, I'm not sure if this is a work-item that's already in-progress by somebody. I noticed that weston has a headless compositor backend and a no-op renderer, which effectively do not depend on having OpenGL or *a* windowing system available at the time that weston is run. It would be really great if we could run the tests against this backend, because then it means that they can be run on foreign machines like continuous integration servers with ease. I've tried quickly changing the backend over in weston-test-env - currently most of the tests are failing because client-pointer is not initialized. Upon initializing that, most of them pass except those that depend on client-keyboard . I suspect that this will involve having to create a wl_keyboard with a set of dummy xkb data for layouts, though I'm not sure how to do that yet. My question to the list is - does it make sense to submit a patch to change the tests over to use the headless backend? Or is the preference to keep them using the wayland or x11 backend for now. Best, Sam. -- Sam Spilsbury ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel -- Sam Spilsbury -- Sam Spilsbury ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/wayland-devel
