That's strange ... Hudson lists a user "Peter Firmstone" with last activity 4 days ago. The user that was setup for me last year unfortunately disappeared from the list :-(
2010/8/27 Peter Firmstone <[email protected]> > Ok, sounds good, I don't have access to Hudson unfortunately though, can > anyone assist here? > > > Jonathan Costers wrote: > >> Ideally it should, yes. So we get visibility on failures sooner ... >> >> To accomplish that, we need to make a change to the property value of >> run.categories and add the missing categories (I've already added two more >> the last cpl days). >> However, adding all other categories to the run.categories property will >> make our next Hudson builds take a very, very long time. >> >> So, as proposed above, we could (request to) setup a separate Hudson job >> that does this complete test run nightly, instead of triggered by SVN >> changes. >> - the current Hudson job would build triggered by SVN changes and run some >> unit tests and/or some selected QA tests (fast) >> - a new Hudson job would do nighly (or otherwise scheduled) builds and run >> the whole QA suite (very slow) >> >> How does that sound to you? >> >> Speaking of Hudson, any chance we can get that up and running again? I see >> nothing has been built in a while. >> >> 2010/8/27 Peter Firmstone <[email protected]> >> >> >> >>> Ok, I'm happy if we do that then, I just want to make sure I'm running >>> all >>> the tests I should, that's all, so qa.run should run all categories by >>> default? >>> >>> Peter. >>> >>> >>> Jonathan Costers wrote: >>> >>> >>> >>>> There is one now already: qa.run >>>> We just need to pass it more test category names (property >>>> run.categories) >>>> by default. >>>> Maybe I am misunderstanding you? >>>> >>>> 2010/8/27 Peter Firmstone <[email protected]> >>>> >>>> >>>> >>>> >>>> >>>>> Hi JC, >>>>> >>>>> Can we have an ant target for running all the tests? >>>>> >>>>> And how about a qa.run.hudson target? >>>>> >>>>> I usually use run-categories, to isolate what I'm working on, but we >>>>> definitely need a target that runs everything that should be, even if >>>>> it >>>>> does take overnight. >>>>> >>>>> Regards, >>>>> >>>>> Peter. >>>>> >>>>> >>>>> Jonathan Costers wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> 2010/8/24 Patricia Shanahan <[email protected]> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On 8/22/2010 4:57 PM, Peter Firmstone wrote: >>>>>>> ... >>>>>>> >>>>>>> Thanks Patricia, that's very helpful, I'll figure it out where I >>>>>>> went >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> wrong this week, it really shows the importance of full test >>>>>>>> coverage. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> ... >>>>>>> >>>>>>> I strongly agree that test coverage is important. Accordingly, I've >>>>>>> done >>>>>>> some analysis of the "ant qa.run" output. >>>>>>> >>>>>>> There are 1059 test description (*.td) files that exist, and are >>>>>>> loaded >>>>>>> at >>>>>>> the start of "ant qa.run", but that do not seem to be run. I've >>>>>>> extracted >>>>>>> the top level categories from those files: >>>>>>> >>>>>>> constraint >>>>>>> discoveryproviders_impl >>>>>>> discoveryservice >>>>>>> end2end >>>>>>> eventmailbox >>>>>>> export_spec >>>>>>> io >>>>>>> javaspace >>>>>>> jeri >>>>>>> joinmanager >>>>>>> jrmp >>>>>>> loader >>>>>>> locatordiscovery >>>>>>> lookupdiscovery >>>>>>> lookupservice >>>>>>> proxytrust >>>>>>> reliability >>>>>>> renewalmanager >>>>>>> renewalservice >>>>>>> scalability >>>>>>> security >>>>>>> start >>>>>>> txnmanager >>>>>>> >>>>>>> I'm sure some of these tests are obsolete, duplicates of tests in >>>>>>> categories that are being run, or otherwise inappropriate, but there >>>>>>> does >>>>>>> seem to be a rich vein of tests we could mine. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> The QA harness loads all .td files under the "spec" and "impl" >>>>>> directories >>>>>> when starting and only witholds the ones that are tagged with the >>>>>> categories >>>>>> that we specify from the Ant target. >>>>>> Whenever a test is really obsolete or otherwise not supposed to run, >>>>>> it >>>>>> is >>>>>> marked with a "SkipTestVerifier" in its .td file. >>>>>> Most of these are genuine and should be run though. >>>>>> There are more categories than the ones you mention above, for >>>>>> instance: >>>>>> "spec", "id", "id_spec", etc. >>>>>> Also, some tests are tagged with multiple categories and as such >>>>>> duplicates >>>>>> can exist when assembling the list of tests to run. >>>>>> >>>>>> The reason not all of them are run (by Hudson) now is that we give a >>>>>> specific set of test categories that are known (to me) to run >>>>>> smoothly. >>>>>> There are many others that are not run (by default) because issue(s) >>>>>> are >>>>>> present with one or more of the tests in that category. >>>>>> >>>>>> I completely agree with the fact that we should not exclude complete >>>>>> test >>>>>> categories because of one test failing. >>>>>> What we probably should do is tag any problematic test (due to >>>>>> infrastructure or other reasons) with a SkipTestVerifier for the time >>>>>> being >>>>>> so that it is not taken into account by the QA harness for now. >>>>>> That way, we can add all test categories to the default Ant run. >>>>>> However, this would take a large amount of time to run (I've tried it >>>>>> once, >>>>>> and killed the process after several days), which brings us to your >>>>>> next >>>>>> point: >>>>>> >>>>>> Part of the problem may be time to run the tests. I'd like to propose >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> splitting the tests into two sets: >>>>>>> >>>>>>> 1. A small set that one would run in addition to the relevant tests, >>>>>>> whenever making a small change. It should *not* be based on skipping >>>>>>> complete categories, but on doing those tests from each category that >>>>>>> are >>>>>>> most likely to detect regression, especially regression due to >>>>>>> changes >>>>>>> in >>>>>>> other areas. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Completely agree. However, most of the QA tests are not clear unit or >>>>>> regression tests. They are more integration/conformance tests that >>>>>> test >>>>>> the >>>>>> requirements of the spec and its implementation. >>>>>> Identifying the list of "right" tests to run as part of the small set >>>>>> you >>>>>> mention would require going through all 1059 test descriptions and >>>>>> their >>>>>> sources. >>>>>> >>>>>> 2. A full test set that may take a lot longer. In many projects, there >>>>>> is >>>>>> a >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> "nightly build" and a test sequence that is run against that build. >>>>>>> That >>>>>>> test sequence can take up to 24 hours to run, and should be as >>>>>>> complete >>>>>>> as >>>>>>> possible. Does Apache have infrastructure to support this sort of >>>>>>> operation? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Again, completely agree. I'm sure Apache supports this through Hudson. >>>>>> We >>>>>> could request to setup a second build job, doing nightly builds and >>>>>> running >>>>>> the whole test suite. Think this is the only way to make running the >>>>>> complete QA suite automatically practical. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Are there any tests that people *know* should not run? I'm thinking >>>>>>> of >>>>>>> running the lot just to see what happens, but knowing ones that are >>>>>>> not >>>>>>> expected to work would help with result interpretation. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> See above, tests of that type should have already been tagged to be >>>>>> skipped >>>>>> by the good people that donated this test suite. >>>>>> I've noticed that usually, when a SkipTestVerifier is used in a .td >>>>>> file, >>>>>> someone has put some comments in there to explain why it was tagged as >>>>>> such. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Patricia >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > >
