Actually ... rephrase that ... anyone disagree? From: Richard Brown (gmail) Sent: Wednesday, April 27, 2011 10:17 AM To: [email protected] Subject: Re: [nhibernate-development] Re: My feeling of our Tests
I guess I’m just stating the obvious, but does that mean we agree (generally) not to make this any worse then? i.e., no more generative tests, and instead focused small tests. The obvious concern being that if we spend the effort putting some of the ‘slow’ tests into a daily build, then this is seen as acceptable and expands over time (leading to further ‘slow’ tests and potentially daily broken builds that are hard to repair). From: Stephen Bohlen Sent: Tuesday, April 26, 2011 9:44 PM To: [email protected] Subject: Re: [nhibernate-development] Re: My feeling of our Tests Haha-- that's what you get for taking time 'off' :) I would certainly support a strategy of moving slower-performing tests into a 'daily build' suite, but only as a final possible fall-back approach b/c that would mean we'd eventually begin to lose "if the CI build is green, we can safely assume the code is OK to check-out and work with" (which I feel to be a pretty valuable thing to be able to say). I'm heartened by the fact that just with some minor tweaks Fabio was able to cut the duration of these tests just about in half already (without needing to reduce their scope, effectiveness, etc. in any way) so I am further hopeful that other optimizations may also be possible to discover with further investigation. If there is (as there seems to be) general agreement that these tests are valuable and necessary, then we obviously need to find a way to "live with them" such that they don't become a burden in one area while trying to solve problems in another. The silver bullet here is to just eliminate all these tests, but since we don't want to do that (and I agree!), the test duration will probably have to be mitigated by a large number of minor changes to the tests, their assertions, the size of the test data, and other factors, each of which is unlikely to solve the issue is isolation but all of which together stand a chance of further reducing the time needed to execute this test suite. FWIW, I still remain intensely skeptical that 1008 and 4608 entities are necessary to validate the proper behavior of the LINQ provider, notwithstanding the fact that some test generator out there seems to disagree with me :) Steve Bohlen [email protected] http://blog.unhandled-exceptions.com http://twitter.com/sbohlen On Tue, Apr 26, 2011 at 4:20 PM, Patrick Earl <[email protected]> wrote: Wow, was busy and missed the whole discussion. :) My general feeling is that full test suites like this are: * Easier to write (saving precious time) * Complete (at least within the bounds of what they're testing) * Time consuming to run Obviously the first two are desirable properties and having complete testing is at times very useful to ensure the correct operation of code when working with it as a black box. The time consumed to run these automated tests is problematic though, hence I would suggest that splitting the tests into a set of "instant tests" and then a separate set of "full coverage" tests. The full coverage tests would be handled by the build server (if long enough, even potentially just once per day). The "instant tests" would be run all the time as usual. My general feeling is that NHibernate is under-tested and that it would be a good idea to allow for larger test suites as things continue to evolve. Patrick Earl
<<wlEmoticon-winkingsmile[1].png>>
