Re: Some shortcomings in gtestutils
On 21.02.2013 03:46, Federico Mena Quintero wrote: Hi, everyone, I've been writing some tests for GtkFileChooserButton and putting them in gtk+/gtk/tests/filechooser.c - this is the old test suite, resurrected and alive. So, I've been learning what gtestutils provides. It is not bad, but it seems pretty pedestrian on some counts. These are some things I'd like to change, or at least for someone to point me in the right way of doing them: * Tests abort as soon as a test fails, since tests ultimately depend on variations of g_assert(). This is fine in that it makes the ensure things are okay code look the same as generic sanity check code. However, it also means that a test suite aborts as soon as it encounters the first test that doesn't pass. There is g_test_fail(), but the documentation pretty much only says, You can use this function if your test failed in a recoverable way. I don't know if that means that the test couldn't find a data file (but can continue anyway), or if it means call this if you want the test to fail but want the test suite to continue running. In gstreamer (and also in buzzard) we use check.sf.net. This (optionally) forks tests. Both projects have a couple of utilities around it (like env-vars to run a specific suite/testcase, handy functions/macros for assertions, support for glib specific types). Would be nice to pull the nice parts out and get them into gtest. Stefan * It's hard to get a debugger up in the middle of make check. I can't think of a way to do it other than manually inserting code to sleep in the middle of the faulty test, and then attaching with a debugger manually. Maybe having an environment variable or something, so that I could run $ G_PAUSE_ON_FAILED_TESTS=1 make check when a test fails, I would get told, attach to $pid now or somthing. Maybe even have a G_NAME_OF_TEST_TO_PAUSE_IN variable to pause for a specific test, not any one that fails. * The documentation on gtestutils is far from stellar :) * Now that the a11y infrastructure is included in-the-box with GTK+, it sounds like a good idea to start putting an a11y-based testing infrastructure in GTK+ itself. For the file chooser's tests, I had to do some manual hacks to simulate, click this button, find the dialog, click the dialog's button... this is tricky C code that assumes too much knowledge of the internals of the toolkit, and I'd love to write some higher-level stuff instead for such tests. (Feel free to reply about this in a different thread - this could derail *this* thread pretty fast) :) Thoughts? We have a useful battery of tests now, and it only seems that improving the testing infrastructure could lead to people actually wanting to write more test code. Federico ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
On Thu, 2013-02-21 at 11:12 +, Philip Withnall wrote: Add the following in your Makefile.am: TESTS_ENVIRONMENT = libtool --mode=execute gdb Then run `make check` as normal, and magically all of your tests will be run under gdb. If you just want to run a specific test binary under gdb, use `make check TESTS=my-test-binary` Damn, this is why I love you, people. Thanks! Clearly I have to read the automake manual more thoroughly... Federico ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
On Wed, 2013-02-20 at 20:46 -0600, Federico Mena Quintero wrote: * It's hard to get a debugger up in the middle of make check. I can't think of a way to do it other than manually inserting code to sleep in the middle of the faulty test, and then attaching with a debugger manually. Maybe having an environment variable or something, so that I could run Add the following in your Makefile.am: TESTS_ENVIRONMENT = libtool --mode=execute gdb Then run `make check` as normal, and magically all of your tests will be run under gdb. If you just want to run a specific test binary under gdb, use `make check TESTS=my-test-binary` Philip signature.asc Description: This is a digitally signed message part ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
hi Federico; On 21 February 2013 02:46, Federico Mena Quintero feder...@gnome.org wrote: So, I've been learning what gtestutils provides. It is not bad, but it seems pretty pedestrian on some counts. These are some things I'd like to change, or at least for someone to point me in the right way of doing them: * Tests abort as soon as a test fails, since tests ultimately depend on variations of g_assert(). This is fine in that it makes the ensure things are okay code look the same as generic sanity check code. However, it also means that a test suite aborts as soon as it encounters the first test that doesn't pass. There is g_test_fail(), but the documentation pretty much only says, You can use this function if your test failed in a recoverable way. I don't know if that means that the test couldn't find a data file (but can continue anyway), or if it means call this if you want the test to fail but want the test suite to continue running. Matthias, Colin, Dan, and I were discussing this in the bug to add TAP support for GTest, to get decent report harnesses instead of our homegrown test report XML hack: https://bugzilla.gnome.org/show_bug.cgi?id=692125 in short, the only way to safely run these tests in C is to execute each unit inside a separate process; it's not a trivial change, though. for Clutter, we do compile all units inside the same binary (to cut down compilation and link time), and then execute each unit independently; we still abort() as soon as one unit asserts, but we can easily change that and keep running. I also added a skip() and a todo() wrappers, but that was just for my convenience. * It's hard to get a debugger up in the middle of make check. I can't think of a way to do it other than manually inserting code to sleep in the middle of the faulty test, and then attaching with a debugger manually. Maybe having an environment variable or something, so that I could run $ G_PAUSE_ON_FAILED_TESTS=1 make check you can use TEST_ENVIRONMENT to set up the test environment variables and command line arguments. it's standard automake. I use it to pass the backend to the ABI check script, so that it's aware of which kind of ABI should be available. ciao, Emmanuele. -- W: http://www.emmanuelebassi.name B: http://blogs.gnome.org/ebassi/ ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
Il Thu, 21 Feb 2013 12:31:33 + Simon McVittie simon.mcvit...@collabora.co.uk scrisse: On 21/02/13 11:42, Nicola Fontana wrote: * There is no way to do a make check while cross-compiling, by using wine for example. If you're on a Linux distribution whose Wine packaging sets up binfmt_misc to run Windows executables through Wine (Debian does), then this works: chmod +x notepad.exe ./notepad.exe I was thinking more along the lines of providing out-of-box support from gtester, such as: gtester --launcher=wine mytest and if that works, then in principle so does 'make check'. In practice it probably won't, until you apply suitable workarounds (like scattering $(EXEEXT) throughout the regression tests). Could you elaborate on why do you think it probably wont work? I thought binfmt was a viable work-around. Ciao. -- Nicola ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Some shortcomings in gtestutils
Hi, everyone, I've been writing some tests for GtkFileChooserButton and putting them in gtk+/gtk/tests/filechooser.c - this is the old test suite, resurrected and alive. So, I've been learning what gtestutils provides. It is not bad, but it seems pretty pedestrian on some counts. These are some things I'd like to change, or at least for someone to point me in the right way of doing them: * Tests abort as soon as a test fails, since tests ultimately depend on variations of g_assert(). This is fine in that it makes the ensure things are okay code look the same as generic sanity check code. However, it also means that a test suite aborts as soon as it encounters the first test that doesn't pass. There is g_test_fail(), but the documentation pretty much only says, You can use this function if your test failed in a recoverable way. I don't know if that means that the test couldn't find a data file (but can continue anyway), or if it means call this if you want the test to fail but want the test suite to continue running. * It's hard to get a debugger up in the middle of make check. I can't think of a way to do it other than manually inserting code to sleep in the middle of the faulty test, and then attaching with a debugger manually. Maybe having an environment variable or something, so that I could run $ G_PAUSE_ON_FAILED_TESTS=1 make check when a test fails, I would get told, attach to $pid now or somthing. Maybe even have a G_NAME_OF_TEST_TO_PAUSE_IN variable to pause for a specific test, not any one that fails. * The documentation on gtestutils is far from stellar :) * Now that the a11y infrastructure is included in-the-box with GTK+, it sounds like a good idea to start putting an a11y-based testing infrastructure in GTK+ itself. For the file chooser's tests, I had to do some manual hacks to simulate, click this button, find the dialog, click the dialog's button... this is tricky C code that assumes too much knowledge of the internals of the toolkit, and I'd love to write some higher-level stuff instead for such tests. (Feel free to reply about this in a different thread - this could derail *this* thread pretty fast) :) Thoughts? We have a useful battery of tests now, and it only seems that improving the testing infrastructure could lead to people actually wanting to write more test code. Federico ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
On Thu, Feb 21, 2013 at 10:46 AM, Federico Mena Quintero feder...@gnome.org wrote: Hi, everyone, I've been writing some tests for GtkFileChooserButton and putting them in gtk+/gtk/tests/filechooser.c - this is the old test suite, resurrected and alive. So, I've been learning what gtestutils provides. It is not bad, but it seems pretty pedestrian on some counts. These are some things I'd like to change, or at least for someone to point me in the right way of doing them: Warning, controversial mail ahead. I've noticed that gtester and gtestutils is a little lackluster. I wanted to see what the GNOME community's thoughts on using an established xUnit framework like Google Test, CPPUnit or boost::test would be. I know, its C++. I get that's not appreciated around here. The only reason why I suggest considering frameworks like these is that they're incredibly powerful, easy to use and I've found that they can be applied to almost any situation, C or C++. Google Test addresses some of the concerns listed below, and does a lot more, like custom test environments, SetUp/TearDown fixtures, matchers (which allow you to do something like EXPECT_THAT (value, MatchesSomething ()), where that would be looking inside lists, variants, whatever). It also ties in really nicely with Google Mock, which allows you to define mock implementations of classes and control on a very fine level how those mock classes behave on a per-test basis. I've used Google Mock alongside in my own GObject projects to great effect[1], and I've been playing around with the idea to use GObject-Introspection to automatically generate them from GInterfaces. (Marketspeak: Google Test has some really strong credentials in this area. It is the test framework of choice for projects like Chromium, XBMC, Xorg and projects at Canonical, such as Unity, Nux and Compiz. Chromium in particular has tens of thousands of unit, integration, system and acceptance tests, mostly all written using Google Test. Self-promotion: Compiz has over 1350 Unit, Integration and Acceptance tests, and works very well with Google Test). Its just some food for thought - I agree with Federico that having a flexible, powerful and easy to use test framework certainly lowers a substantial barrier to entry . *snip* I think that some of the ideas you've raised here are excellent. To address some of your concerns: 1. Most xUnit frameworks I know of have something like ASSERT_* and EXPECT_*. The former will set the test to failed and return directly. The latter will just set the test to failed and continue. Generally speaking, it is acceptable to have multiple ASSERT_ statements because they usually belong in SetUp/TearDown logic. ASSERT_ usually means this test failed because it could not be run due to a failed precondition in SetUp. Ideally, every test should have only one EXPECT_* statement. The EXPECT_* statement is the essence of the test, and tests should test one thing so that you have pinpoint resolution as to which part of the unit failed. 2. The best way to handle this case is to expose the test binary in the build-directory so that users can run it directly. Sometimes you might have multiple test binaries, but that is fine. Ideally no test should have any dependency on the previous test finishing. If that happens, then you've got a serious problem with your code. 3. This is a good point 4. This is an excellent idea for bootstrapping an acceptance testing framework. Care should always be taken when writing acceptance tests though, in particular: a. They aren't a replacement for unit or integration tests. They run particularly slowly, and are usually more failure prone because they can be impacted by external factors that you might not expect. The best kind of code coverage is code covered at a unit, integration and acceptance level. b. Acceptance testing can also be tricky because they often rely on introspection through, eg, the a11y interface. That can create unwanted coupling between the internals of your system and the test, which means that you'll be constantly adjusting the tests as the code is adjusted. Determine what kind of interface you want to expose for test verification and make the tests rely on that. c. Running on some kind of dummy display server (eg, wayland, xorg with the dummy video driver[2]) is always a good idea because it means that you don't have awkward situations where you can't get tests running in continuous integration servers. Tests that aren't run are broken tests. d. Never ever ever ever ever ever ever ever ever rely on sleep()s, timing, or whatever in the tests when verifying conditions. Autopilot in Unity does this and it is a giant fail. Otherwise, great to see these topics are being talked about. [1] http://bazaar.launchpad.net/~compiz-team/compiz/0.9.9/files/head:/gtk/window-decorator/tests/ [2] Xorg-GTest is good for this. Federico ___ gtk-devel-list
Re: GUI Testing (was: Some shortcomings in gtestutils)
On Feb 20, 2013, at 6:46 PM, Federico Mena Quintero feder...@gnome.org wrote: * Now that the a11y infrastructure is included in-the-box with GTK+, it sounds like a good idea to start putting an a11y-based testing infrastructure in GTK+ itself. For the file chooser's tests, I had to do some manual hacks to simulate, click this button, find the dialog, click the dialog's button... this is tricky C code that assumes too much knowledge of the internals of the toolkit, and I'd love to write some higher-level stuff instead for such tests. (Feel free to reply about this in a different thread - this could derail *this* thread pretty fast) :) There are two ways to go about GUI testing. One is to retrieve the coordinates of a control and inject a mouse or keyboard event at those coordinates, the other is to give every control an id that can be used to send events to. The first is incredibly brittle and the second is AFAIK supported only by wxWidgets. [1] ISTR that there was an implementation of the coordinate-tracking sort for Gtk1, but I've lost track of it. It would be a pretty big change to Gtk to introduce control ids and event injectability. Regards, John Ralls [1] http://wxguitest.sourceforge.net/ ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
On Thu, Feb 21, 2013 at 1:17 PM, Peter Hutterer peter.hutte...@who-t.net wrote: Having worked with googletest and xorg-gtest [1] for X integration testing, I can say the most annoying bit is to get the whole thing to compile. The C++ ODR prevents us from building gtest and xorg-gtest as library and then compiling against that. and autotools is not happy with having external source files. if you're planning to share a test framework across multiple source repositories, that can be a major pain. [1] http://cgit.freedesktop.org/~whot/xorg-integration-tests/ I agree, this is the one drawback of google-test. *snip* fwiw, one of the drawbacks I found with the multiple binary case is that it reduces the chance of running all tests every time. there's a sweet spot somewhere between too many and too few binaries and I suspect it differs for each project. A good way to handle that is to have a separate test-runner that runs the make check/test target. That can usually just be a makefile rule or something else (in compiz we use ctest) for the googletest case for example separate binaries will give you a separate junit xml output, which make some regression comparisons harder. We ran into this problem as well. I think the solution was two fold: 1. First of all, we wanted to be able to introspect test binaries so that the test runner would be able to show each individual one. [1] is a massive hack, but it works. 2. Second of all, we had to patch google-test to shut up about skipped tests in the junit.xml so that Jenkins didn't think you had like 600,000 tests or something. I can provide this patch upon request, its just somewhere in the midsts of the Canonical Jenkins deployment. [1] http://bazaar.launchpad.net/~compiz-team/compiz/0.9.9/view/head:/cmake/src/compiz/compiz_discover_gtest_tests.cpp -- Sam Spilsbury ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Some shortcomings in gtestutils
On Thu, Feb 21, 2013 at 02:39:21PM +0800, Sam Spilsbury wrote: On Thu, Feb 21, 2013 at 1:17 PM, Peter Hutterer peter.hutte...@who-t.net wrote: Having worked with googletest and xorg-gtest [1] for X integration testing, I can say the most annoying bit is to get the whole thing to compile. The C++ ODR prevents us from building gtest and xorg-gtest as library and then compiling against that. and autotools is not happy with having external source files. if you're planning to share a test framework across multiple source repositories, that can be a major pain. [1] http://cgit.freedesktop.org/~whot/xorg-integration-tests/ I agree, this is the one drawback of google-test. *snip* fwiw, one of the drawbacks I found with the multiple binary case is that it reduces the chance of running all tests every time. there's a sweet spot somewhere between too many and too few binaries and I suspect it differs for each project. A good way to handle that is to have a separate test-runner that runs the make check/test target. That can usually just be a makefile rule or something else (in compiz we use ctest) yeah, I've hooked it up to make check, but that has other issues too. I should really write a script for that. fwiw, we still have multiple binaries (evdev, synaptics, server, libXi, etc.) But initially I had multiple binaries for the server to test various server features (grab, multihead, touch, etc.). Since the features aren't clear-cut though (where do you put touch grab tests?) I found having a single server binary was better. This is what I meant with there being a sweet spot for each project that needs to be found. for the googletest case for example separate binaries will give you a separate junit xml output, which make some regression comparisons harder. We ran into this problem as well. I think the solution was two fold: 1. First of all, we wanted to be able to introspect test binaries so that the test runner would be able to show each individual one. [1] is a massive hack, but it works. 2. Second of all, we had to patch google-test to shut up about skipped tests in the junit.xml so that Jenkins didn't think you had like 600,000 tests or something. I can provide this patch upon request, its just somewhere in the midsts of the Canonical Jenkins deployment. oh, right. that hasn't been a problem for me so far, the jenkins install always runs all tests. the tricky bit for me was tracking which tests are supposed to fail (e.g. on RHEL6 tests for newer features are known-to-fail). so far I'm tracking this largely manually, but the known-to-fail case shouldn't be much of an use-case for an upstream project. Cheers, Peter [1] http://bazaar.launchpad.net/~compiz-team/compiz/0.9.9/view/head:/cmake/src/compiz/compiz_discover_gtest_tests.cpp -- Sam Spilsbury ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
