In my opinion, the tests are independent from each other. So every test
start up with its own environment, and release the resources when finished.
In this way, there's no dependancy chain and no mess at all.
Yes, we've met serveral side effect testcases, how about making a list to
warn the developer? I have some of them:
・set SecurityManager and always set it back, and at this time it should
prove RuntimePermission("setSecurityManager")
・setup a server, bind it to a address, but does not close if test fails, so
add a "try...finally"
・so is other network service
・and file read/write
・more ...2006/4/11, George Harley <[EMAIL PROTECTED]>: > > Geir Magnusson Jr wrote: > > > > > > George Harley wrote: > >> Hi Geir, > >> > >> From my point of view the answer is yes, I ran the tests for the > >> patched module. The build/test server picked up the downstream > >> breakages in the other modules. I fixed matters so that the builds > >> ran again. > > > > Right - I thought we considered the build/test server a safety to keep > > accidents from getting away from us, but given the delay and the > > distribution of people throughout the planet working on this (rather > > than all sitting in the same office), we want to avoid breakages at > > all costs. (Again, I'm guilty of the same thing...) > > > >> > >> Elsewhere in this thread I am trying to cooperate with others to make > >> it simpler to run more - just the necessary - tests prior to making a > >> commit. > > > > I guess the question is "what is necessary"? Given a dependency chain > > > > A <- B <- C > > > > I don't think it's only the immediate dependencies of a given module > > A, but rather the full dependency graph, because a "1 away" module (C) > > might use a direct dependency (B) in a way different than the test > > cases for that direct dependency do (especially since we aren't quite > > up to speed with doing implementation testing...) > > Hi Geir, > > Sure, point taken. Testing A and then its set of immediate dependants > (just B in this example) will greatly reduce the risk of downstream > module breakages but will not completely catch all such events. > Actually, subject to the quality of the tests in C, it is possible that > even testing A, B and C will not find any such submerged problems. > > I am not an expert in the theory of testing ( as if you had to be told > :-) ) but instinct alone tells me that making it convenient and quick to > run a module's tests and those of its dependants will definitely > *improve* matters - it would certainly have helped me last night before > I broke the build. > > Best regards, > George > > > > >> > >> Thank you for your support. > >> > > > > Um... > > geir > > > >> > >> Best regards, > >> George > >> > >> > >> Geir Magnusson Jr wrote: > >>> So the answer is "no"? > >>> > >>> George Harley wrote: > >>>> Geir Magnusson Jr wrote: > >>>>> Just curious (and this isn't a criticism - I'm just as guilty of > >>>>> not doing this)... > >>>>> > >>>>> Don't you run the tests before committing? > >>>> > >>>> Hi Geir, > >>>> > >>>> Depends what do you mean by "the tests". > >>>> > >>>> The change was completely encapsulated in the text component. I ran > >>>> the text tests. I checked that the new tests introduced in the > >>>> issue patch gave the same results when run against a 5.0 RI. All > >>>> was fine. > >>>> > >>>> The automated build and test server (which runs all the tests in > >>>> all the modules) found the problem. That is it's job and it did it > >>>> well. The problem was found relatively quickly and has now been > >>>> remedied. The turnaround time would have been quicker if the > >>>> automated build and test server issued a failure notice on failing > >>>> tests but that is now in the process of getting sorted out. > >>>> > >>>> IMHO this problem was a bit of an odd problem as the failures did > >>>> not arise in the patched module's test suite but in other modules. > >>>> That is a good indication that the test suite for text is in need > >>>> of some scrutiny. That is a useful thing to be made aware of. > >>>> > >>>> > >>>> Best regards, > >>>> George > >>>> > >>>>> > >>>>> geir > >>>>> > >>>>> George Harley wrote: > >>>>>> Hi, > >>>>>> > >>>>>> It *seems* like things started failing after I committed the > >>>>>> changes for HARMONY-205 last night. I'm looking into this now. If > >>>>>> the investigation begins to take up too much time I will back the > >>>>>> changes out. > >>>>>> > >>>>>> Best regards, > >>>>>> George > >>>>>> > >>>>>> > >>>>>> Stepan Mishura wrote: > >>>>>>> The same for me + DatagramChannelTest > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Stepan. > >>>>>>> > >>>>>>> On 4/11/06, Mark Hindess wrote: > >>>>>>> > >>>>>>>> No. These: > >>>>>>>> > >>>>>>>> F > >>>>>>>> > org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder > >>>>>>>> > >>>>>>>> E > >>>>>>>> > org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01 > >>>>>>>> > >>>>>>>> E > >>>>>>>> > org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02 > >>>>>>>> > >>>>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt > >>>>>>>> E > >>>>>>>> > org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode > >>>>>>>> > >>>>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid > >>>>>>>> F java.security.cert.X509CertSelectorTest.testMatch > >>>>>>>> F java.security.cert.X509CertSelectorTest.testClone > >>>>>>>> F tests.api.java.sql.DateTest.testSetTimelong > >>>>>>>> F tests.api.java.sql.DateTest.testToString > >>>>>>>> F tests.api.java.sql.DateTest.testValueOf > >>>>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint > >>>>>>>> F tests.api.java.sql.TimestampTest.testToString > >>>>>>>> F tests.api.java.util.DateTest.test_toGMTString > >>>>>>>> F tests.api.java.util.DateTest.test_toString > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Mark. > >>>>>>>> > >>>>>>>> On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote: > >>>>>>>>> On 4/11/06, Mark Hindess wrote: > >>>>>>>>>> Yes. I was using the failureproperty mechanism. Trying to get > >>>>>>>>>> this > >>>>>>>>>> property propogated back to the top level ant file was what I > >>>>>>>>>> was > >>>>>>>>>> having trouble with. > >>>>>>>>>> > >>>>>>>>>> Using a file as you suggest might help. I'll give that a try > >>>>>>>> shortly... > >>>>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors on r393111. > >>>>>>>>> I guess that you have 9 tests from DatagramChannelTest passed. > >>>>>>>>> And 12 + > >>>>>>>> 3 = > >>>>>>>>> 15 :-) > >>>>>>>>> > >>>>>>>>> (And > >>>>>>>>>> there are typos - "mathc" should be "match" - in the failure > >>>>>>>>>> messages > >>>>>>>>>> for java.security.cert testMatch and testClone.) > >>>>>>>>> I've fixed typo. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Stepan. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>>> Mark. > >>>>>>>>>> > >>>>>>>>>> On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote: > >>>>>>>>>>> On 4/11/06, Mark Hindess wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Stepan, > >>>>>>>>>>>> > >>>>>>>>>>>> I have another build running (but without notifications > >>>>>>>>>>>> going to > >>>>>>>> the > >>>>>>>>>>>> list) that runs: > >>>>>>>>>>>> > >>>>>>>>>>>> 1) build (with reference jdk) > >>>>>>>>>>>> 2) build with what we created with 1) > >>>>>>>>>>>> 3) test > >>>>>>>>>>>> 4) create classlists and compare with class load data for > >>>>>>>> applications > >>>>>>>>>>>> (tomcat, geronimo, continuum, etc.) > >>>>>>>>>>>> 5) generate JAPI results > >>>>>>>>>>>> > >>>>>>>>>>>> I'd like to fail this build at step 3, but I can't see an > >>>>>>>>>>>> easy way > >>>>>>>> to > >>>>>>>>>>>> get 'ant -f make/build.xml test' to run all tests and then > >>>>>>>>>>>> fail if > >>>>>>>> any > >>>>>>>>>>>> of the module test sub-targets had test failures. I could > >>>>>>>>>>>> parse > >>>>>>>> the > >>>>>>>>>>>> output I suppose, but I'd rather get ant to propagate the > >>>>>>>>>>>> junit > >>>>>>>> tasks > >>>>>>>>>>>> failure property back up to the top level. I've tried a > >>>>>>>>>>>> couple of > >>>>>>>>>>>> things and none seem to work. Any suggestions welcome. > >>>>>>>>>>> Mark, did you try failureproperty parameter for junit task? > >>>>>>>>>>> We may add it to ant sub-targets to raise a flag, for example, > >>>>>>>> create > >>>>>>>>>> file " > >>>>>>>>>>> TESTS.FAILED" in the root, when tests for some module fail. > >>>>>>>>>>> And in > >>>>>>>> the > >>>>>>>>>> end > >>>>>>>>>>> of tests suite run check whether this file exists on not. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Stepan. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>>> Mark. > >>>>>>>>>>>> > >>>>>>>>>>>> On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote: > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I've checked out at morning last updates, built the code > base > >>>>>>>> and > >>>>>>>>>> run > >>>>>>>>>>>> the > >>>>>>>>>>>>> tests …and there are 24 tests failures! > >>>>>>>>>>>>> > >>>>>>>>>>>>> There are 9 tests failures in > >>>>>>>>>>>>> > org.apache.harmony.tests.java.nio.channels.DatagramChannelTest > >>>>>>>>>>>>> �C > >>>>>>>> I > >>>>>>>>>> saw > >>>>>>>>>>>> these > >>>>>>>>>>>>> failures before from time to time. It seems that tests > >>>>>>>>>>>>> depend on > >>>>>>>>>> some > >>>>>>>>>>>> race > >>>>>>>>>>>>> conditions because they may pass if I rerun them. Paulex, > are > >>>>>>>> these > >>>>>>>>>>>> tests > >>>>>>>>>>>>> passing for you? > >>>>>>>>>>>>> > >>>>>>>>>>>>> And there are new 15 test failures. So now if I'll make a > >>>>>>>>>>>>> code > >>>>>>>>>> update > >>>>>>>>>>>> or a > >>>>>>>>>>>>> bug fix how I can be 100% sure that I don't do any > >>>>>>>>>>>>> regression? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Currently if a commit breaks the Harmony classlib build a > >>>>>>>>>> notification > >>>>>>>>>>>> with > >>>>>>>>>>>>> subject: "[continuum] BUILD FAILURE: Classlib/linux.ia32" > >>>>>>>>>>>>> will > >>>>>>>> be > >>>>>>>>>> send > >>>>>>>>>>>> to > >>>>>>>>>>>>> harmony-commits mailing list. Is it possible to have the > same > >>>>>>>> for > >>>>>>>>>> tests? > >>>>>>>>>>>> I > >>>>>>>>>>>>> mean that after completing automatic build the existing > >>>>>>>>>>>>> classlib > >>>>>>>>>> tests > >>>>>>>>>>>> suite > >>>>>>>>>>>>> should be run. If there are failing tests then a report > >>>>>>>> notification > >>>>>>>>>>>> with > >>>>>>>>>>>>> corresponding subject will be send. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Stepan Mishura > >>>>>>>>>>>>> Intel Middleware Products Division > >>>>>>>>>>>> -- > >>>>>>>>>>>> Mark Hindess <[EMAIL PROTECTED]> > >>>>>>>>>>>> IBM Java Technology Centre, UK. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > --------------------------------------------------------------------- > >>>>>>>> > >>>>>>>>>>>> Terms of use : > >>>>>>>>>>>> http://incubator.apache.org/harmony/mailing.html > >>>>>>>>>>>> To unsubscribe, e-mail: > >>>>>>>> [EMAIL PROTECTED] > >>>>>>>>>>>> For additional commands, e-mail: > >>>>>>>> [EMAIL PROTECTED] > >>>>>>>>>>> -- > >>>>>>>>>>> > >>>>>>>> > --------------------------------------------------------------------- > >>>>>>>> > >>>>>>>>>>> Terms of use : > http://incubator.apache.org/harmony/mailing.html > >>>>>>>>>>> To unsubscribe, e-mail: > >>>>>>>>>>> [EMAIL PROTECTED] > >>>>>>>>>>> For additional commands, e-mail: > >>>>>>>> [EMAIL PROTECTED] > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Stepan Mishura > >>>>>>>>>>> Intel Middleware Products Division > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Mark Hindess <[EMAIL PROTECTED]> > >>>>>>>>>> IBM Java Technology Centre, UK. > >>>>>>>>>> > >>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>> > >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>>>>> To unsubscribe, e-mail: > >>>>>>>>>> [EMAIL PROTECTED] > >>>>>>>>>> For additional commands, e-mail: > >>>>>>>>>> [EMAIL PROTECTED] > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>> > >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>>>> To unsubscribe, e-mail: > >>>>>>>>> [EMAIL PROTECTED] > >>>>>>>>> For additional commands, e-mail: > >>>>>>>>> [EMAIL PROTECTED] > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Stepan Mishura > >>>>>>>>> Intel Middleware Products Division > >>>>>>>>> > >>>>>>>>> > >>>>>>>> -- > >>>>>>>> Mark Hindess <[EMAIL PROTECTED]> > >>>>>>>> IBM Java Technology Centre, UK. > >>>>>>>> > >>>>>>>> > --------------------------------------------------------------------- > >>>>>>>> > >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>>> To unsubscribe, e-mail: > >>>>>>>> [EMAIL PROTECTED] > >>>>>>>> For additional commands, e-mail: > >>>>>>>> [EMAIL PROTECTED] > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> > --------------------------------------------------------------------- > >>>>>>> > >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>> To unsubscribe, e-mail: > >>>>>>> [EMAIL PROTECTED] > >>>>>>> For additional commands, e-mail: > >>>>>>> [EMAIL PROTECTED] > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Stepan Mishura > >>>>>>> Intel Middleware Products Division > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> > >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>> To unsubscribe, e-mail: > [EMAIL PROTECTED] > >>>>>> For additional commands, e-mail: > >>>>>> [EMAIL PROTECTED] > >>>>>> > >>>>>> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>> For additional commands, e-mail: > >>>>> [EMAIL PROTECTED] > >>>>> > >>>>> > >>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>> For additional commands, e-mail: > [EMAIL PROTECTED] > >>>> > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>> For additional commands, e-mail: [EMAIL PROTECTED] > >>> > >>> > >> > >> > >> --------------------------------------------------------------------- > >> Terms of use : http://incubator.apache.org/harmony/mailing.html > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > >> > >> > > > > --------------------------------------------------------------------- > > Terms of use : http://incubator.apache.org/harmony/mailing.html > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Best Regards! Jimmy, Jing Lv China Software Development Lab, IBM
