Gentlemen, great debate. I will provide a bit more information a bit later as I don't want to answer without fully running thru all of the scenarios that everyone lays out. Adam - thanks for pushing on this - it's needed it for a long time.
Cheers, Tim -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 ----- "Adam Heath" <doo...@brainfood.com> wrote: > David E Jones wrote: > > *** First off Adam: it's great to see you working on this stuff, > the > > testing stuff in general needs a LOT more work and hasn't seen much > > attention, so yeah... this is great. > > Completely automatic tests with no side-effects is what I am striving > for. I believe we are close. > > > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote: > > > >>>> The service engine tests define several different test cases. > The > >>>> second test case does a data load. Then each subsequent test > modifies > >>>> the data state of the previous test. They all have to run in > >>>> sequence, for them to pass. > >>>> > >>>> This means that they are *not* separate test cases, but a single > test > >>>> case. A test case is supposed to be self-contained, not > depending on > >>>> the state of anything else. It's supposed to start from empty, > do > >>>> whatever mutative changes it needs to, without any external > >>>> dependencies. > >>> > >>> Who says? > >>> > >>> Why? > >>> > >>> The change you made doesn't change this, it just creates something > new > >>> called a "test group" that I'm guessing is supposed to group > tests > >>> together than depend on eachother. However, it doesn't make the > test > >>> cases independent. > >>> > >>> Also, why introduce a test group concept when we already have the > >>> test-suite concept? Why not just use that? Run one suite at a time > and > >>> only allow dependencies within that suite. > >> > >> Well, TestRunContainer supports running a singly-defined test case > >> from some testdef.xml file. If done against service tests, then > >> you'll get false positives. > >> > >> I'm fine with not having this <test-group>, so long as we don't > allow > >> for running separate tests, that TestRunContainer currently > allows. > >> > >> Currently, we can run all tests suites inside a component, or one > >> separate test in a component, in whatever test suite it resides > in. > >> We have no way to restrict the lookup code to just a single test > suite. > > > > What does all of this have to do with it? > > > > Are you just saying that we need a way to run one test suite at a > time? > > If we're going to move toward the practice of having dependencies > only > > within a single test suite, then yes, I'd agree this is necessary. > > > > At the minute the practice is to allow dependencies within a > component. > > > >>> Still, this hasn't been discussed. The only thing we've discussed > and > >>> somewhat decided on so far (including the ApacheCon discussions I > was > >>> involved in, which was the same as what was discussed before) is > to only > >>> allow dependencies between test cases within components. > >> > >> Well, in some cases, multiple tests defined inside some > test-suite.xml > >> can *not* run in series, or in parallel. Each defined test case > needs > >> a *clean* database. In other cases, the test cases inside the > >> test-suite.xml *do* require running each in series. > > > > Oh yeah? Where? > > > > Here's the scary thing that I'm starting to believe to be the case: > > because you didn't read and research before starting to change > things > > you didn't try running the tests for a component all at once. You > made > > incorrect assumptions about how they were written and how they were > > meant to be run. > > But I did do some. > > The service tests require running everything in the test suite. That > I am absolutely certain of. > > The other thing we know, is that running some tests together causes > problems. I'm not entirely certain if these same bad tests together > exist in a single test suite, or if different test suites conflict. > > > Your rants and complaints about the tests may be incorrect, and the > > "fixes" you've been making for them may be unnecessary. They are > only > > true if you adopt your assertion that all test cases should be > > independent. Otherwise, they are not true and you're heading down a > road > > that others may not appreciate. > > > > Could we maybe step back and talk about this before you cruise along > too > > much more? > > I still stand by the <test-group> addition; it's a new feature, that > seems usefull. It allows combining separately defined test cases > into > a single whole, without giving them each a separate name. It allows > for code reuse, without having to write some java code, or a > simple-method, to combine them. > > However, what we are heading towards is making the granularity be a > test suite. In that case, the service test definition should be > changed back, as well as changing my recent build.xml and > TestListContainer policy. > > Just to restate, I will change the policy to be each suite gets run > separately.
