I guess it was my impression that people were suggesting that if running tests against a single DB would mean 5 mins for comprehensive test-run, that running it against 5 DBs would mean 5 X 5 mins = 25 mins and that this was tending towards unacceptable delay in feedback (these are NOT actual numbers, just an example calculation).
My thought was that If we aren't presently running all tests against all databases on all commits, then we aren't CURRENTLY being made immediately aware of inadvertent breaking changes in all the DB targets. Scheduling once-per-day test-runs of the entire comprehensive list of DB targets would reduce what's currently a "cross your fingers, hope a lot, and wait for bug report for one of the other providers to surface" to a duration of no longer than 24 hours at most :) (without resulting in a 2x, 4x -- or whatever -- increase in the test-run duration on every commit). I completely agree that in a perfect world we'd run 100% of the tests 100% of the time, but if there's traction for the idea that this is impractical due to test-run duration, I was just suggesting a (possible) middle-ground that might be worth considering as it would be significantly better than the present situation without (all that much) increasing the per-commit test-run length. If this isn't of perceived value and the only useful alternative is to run all the tests all the time on every commit then lets not pursue my suggestion :D Steve Bohlen [email protected] http://blog.unhandled-exceptions.com http://twitter.com/sbohlen On Wed, Jul 27, 2011 at 3:53 PM, Patrick Earl <[email protected]> wrote: > For myself, I think I'd just end up going to the build server and > scheduling the other builds right away anyways. I want to know as > soon as possible if my changes are having a detrimental effect > anywhere throughout the system. > > What is the main concern people have? That the SQL Server build gets > delayed? My understanding is that it's general policy to not commit > things that break that test suite anyways, so I can't imagine it would > make much difference. > > Ideally, everything would just run faster, but if we can't have it run > faster, I'm personally good with it just happening sometime soon after > each commit. It's possible that grouping the commits together by > adding a delay could improve some situations, but in general the > commits to NH aren't that frequent, so I feel that having the system > do a full build after each commit is quite reasonable. > > Patrick Earl > > On Wed, Jul 27, 2011 at 1:45 PM, Stephen Bohlen <[email protected]> wrote: > > Is it feasible to consider decoupling the database-specific tests from > the > > commit-triggered-build altogether and running the database-specific tests > as > > part of either a regularly-scheduled once-per-day build or as an > on-demand > > build? If they increase the length of the CI feedback loop unacceptably, > > this might permit us to both benefit from having them wired up but also > > sidestep the negatives associated with them. > > > > Thoughts on the viability of this approach --? > > > > Steve Bohlen > > [email protected] > > http://blog.unhandled-exceptions.com > > http://twitter.com/sbohlen >
