We should get this integrated into buildtest directly and documented. This after all, is the whole point of buildtest.
-Nathan On Tue, Aug 12, 2008 at 9:40 PM, chunrong lai <[EMAIL PROTECTED]> wrote: > Thanks. > (1) Setting for e-mail notifications are in framework.local.properties as > http://harmony.apache.org/subcomponents/buildtest/howto.html. > > # Setup for mail notifications > framework.parameters.usemail=... > [EMAIL PROTECTED] > [EMAIL PROTECTED] > > One may need to copy mail.jar and activation.jar from > infra/build/libs/cruisecontrol-bin-2.6.2/lib/ to $ANT_HOME/lib. > > (2) After running integrity testing the direcory set in > framework.local.properties > > # Directory to store archived results > framework.parameters.optional.upload.dir=*<results_dir>* > > will take effect thus an archived results will be stored there. The file > name is like > > [EMAIL PROTECTED] infra]$ ls ~/ccresult/build/to_upload/ > autouploading linux_x86.drlvm-test.status.zip > linux_x86.jdktools-test.status.zip > linux_x86.classlib-test.status.zip linux_x86.ehwa-api.status.zip > linux_x86.make-snapshot.status.zip > linux_x86.drlvm-reg-test.status.zip linux_x86.hdk.status.zip > uploadresults.sh > > (3) > http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/tools/integrity/status-table/ > > include the table and perl scripts to generate the integrity testing > result page from above results. > > One needs to specify the result file/web page location and can configure the > included platforms/test suites there to enable it. > > So having a web page to look at every evening can be achieved by > uploading/extracting the results files and running the perl scripts > everyday. I see all required settings have been documented. > (4) We also added local contents to cc-exec-adaptor.xml in BTI for the > auto-uploading/web-page generating. It has not been committed/documented. > > update="true" /> > > + <available property="autouploads" > file="${upload.dir}/autouploading"/> > + <antcall target="-auto-upload"/> > </target> > > + <target name="-auto-upload" if="autouploads"> > + <exec executable="${upload.dir}/uploadresults.sh" > dir="${upload.dir}"> > + </exec> > + </target> > </project> > The scripts use scp to (auto)upload results and ssh to (auto)run above perl > scripts remotely (required keys setting). > > Thanks. > > On 8/13/08, Nathan Beyer <[EMAIL PROTECTED]> wrote: >> >> On Mon, Aug 11, 2008 at 10:16 PM, chunrong lai <[EMAIL PROTECTED] >> >wrote: >> >> > Thanks. >> > I think the standard scripts produce that site although I always added >> > publish/e-mail notification setup before install/run the test suites. >> >> >> Where is it at, I'm not able to find it and I would really like to recreate >> this? How do you get it published to people.apache.org? I don't want an >> email, I just want a web page to look at every evening. >> >> Can you document how you set it up? We need to have this integrity thing >> documented, so anyone can recreate it for the next milestone. >> >> -Nathan >> >> >> > >> > It is understandable that some tests in classlib-test and drlvm-test >> get >> > failing report. We just need to understand the failures. >> > As reported in M6, >> > org.apache.harmony.lang.management.MemoryPoolMXBeanImplTest, >> > >> > >> org.apache.harmony.lang.management.tests.java.lang.management.MemoryPoolMXBeanTest >> > etc fail classlib-test. >> > Also we observed that java.lang.ClassGenericsTest.test_2 sometimes fail >> > drlvm-test. >> > Another timeout issue of drlvm-test comes from >> > thread.SmallStackThreadTest_jit, as discussed in HARMONY-4601, the test >> > case >> > may run very slow when system resource is tight and finally introduce a >> > timeout. Re-running the suite generally fixes the timeout. >> > >> > >> > >> > >> > >> > On 8/12/08, Nathan Beyer <[EMAIL PROTECTED]> wrote: >> > > >> > > It's the tests that are failing - both classlib-test and drlvm-test. >> > > >> > > What's the specific integrity setup that's being used for that site? >> Does >> > > the standard out-of-the-box buildtest scripts produce that site? >> > > >> > > -Nathan >> > > >> > > On Mon, Aug 11, 2008 at 9:05 PM, chunrong lai <[EMAIL PROTECTED]> >> > > wrote: >> > > >> > > > hi, Nathan: >> > > > Steps to run testing cycle are listed in >> > > > http://harmony.apache.org/subcomponents/buildtest/index.html. One >> > > > integrity >> > > > testing example is >> > > > >> http://harmony.apache.org/subcomponents/buildtest/howto.html#Extended. >> > > > What is the error message you met (in what step)? What is the >> > > content >> > > > of the framework.local.properties file? I think that will help to >> > figure >> > > > out >> > > > the problem. >> > > > >> > > > >> > > > On 8/12/08, Nathan Beyer <[EMAIL PROTECTED]> wrote: >> > > > > >> > > > > What's being used to produce the integrity tests? I have an x86_64 >> > > Linux >> > > > > box >> > > > > that I can dedicate to testing. I've tried just using buildtest >> with >> > > > > 'classlib,drlvm,classib-test,drlvm-test', but I haven't gotten a >> > clean >> > > > pass >> > > > > yet. >> > > > > >> > > > > -Nathan >> > > > > >> > > > > On Mon, Aug 11, 2008 at 12:26 PM, chunrong lai < >> > [EMAIL PROTECTED] >> > > > > >wrote: >> > > > > >> > > > > > Hi, all: >> > > > > > >> > > > > > Here is r681495 snapshot testing status: >> > > > > > http://people.apache.org/~chunrong/snapshots/r681495/index.html< >> http://people.apache.org/%7Echunrong/snapshots/r681495/index.html> >> > < >> > > http://people.apache.org/%7Echunrong/snapshots/r681495/index.html> >> > > > < >> > > > > http://people.apache.org/%7Echunrong/snapshots/r681495/index.html >> >. >> > > > > > I am using >> > > > > > two platforms (Linux x86, windows x86_64) at the moment. >> Hopefully >> > we >> > > > > will >> > > > > > have other two platforms in future for M8. Well, although we are >> > > > testing >> > > > > > only two platforms for M7, my experience is that the status for >> > > another >> > > > > two >> > > > > > platforms should be not worse or just include some extra >> > intermittent >> > > > > > errors >> > > > > > which can be investigated in some later stages. >> > > > > > >> > > > > > The following suites passed on Linux x86/Windows x86_64 >> platforms: >> > > Ant >> > > > > > Scenario (or self-hosting), Axis application, Dacapo, DRLVM >> > > regression >> > > > > > tests, Geronimo Unit Tests, Scimark, Tomcat scenario, VTS VM Test >> > > > Suite. >> > > > > > >> > > > > > Most of the failures are known issues (for M6). Although we can >> > > > observe >> > > > > > 15~20 new issues, those issues happen only in 1 platform and they >> > > look >> > > > > more >> > > > > > like the intermittent/timeout issues (less reproducible) to me. >> > > Overall >> > > > I >> > > > > > myself would like to think r681495 is more stable than M6. >> > > > > > >> > > > > > Please add your comments and clarifications (please also see M6 >> > > > testing >> > > > > > results http://people.apache.org/~smishura/r653525/< >> http://people.apache.org/%7Esmishura/r653525/> >> > < >> > > http://people.apache.org/%7Esmishura/r653525/> >> > > > < >> > > > > http://people.apache.org/%7Esmishura/r653525/> >> > > > > > , >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://mail-archives.apache.org/mod_mbox/harmony-dev/200805.mbox/[EMAIL >> PROTECTED] >> > > > > > integrity testing results >> > > > > > http://people.apache.org/~chunrong/harmony-integrity/< >> http://people.apache.org/%7Echunrong/harmony-integrity/> >> > < >> > > http://people.apache.org/%7Echunrong/harmony-integrity/> >> > > > < >> > > > > http://people.apache.org/%7Echunrong/harmony-integrity/>as a >> > > > comparison). >> > > > > > >> > > > > > 1) Classlib: >> > > > > > 1.1) Since r644719 (which committed patch for HARMONY-5684) >> > > > > > >> > > > > >> > org.apache.harmony.lang.management.MemoryPoolMXBeanImplTest >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.lang.management.tests.java.lang.management.MemoryPoolMXBeanTest >> > > > > > >> > > > > > failed in both platforms >> > > > > > >> > > > > > 1.2) Two failures >> > > > > > >> > > > > > org.apache.harmony.luni.tests.java.net.MulticastSocketTest >> > > > (Failed >> > > > > > in windows_x86 running of M6) >> > > > > > tests.api.java.security.PermissionCollectionTest >> > > > (Failed >> > > > > > in M6) >> > > > > > >> > > > > > are observed in Linux x86. >> > > > > > >> > > > > > 2) DRLVM tests: >> > > > > > 2.1) One failure >> > > > > > >> > > > > > java.lang.ClassGenericsTest.test_2 >> > > > > > >> > > > > > is observed in Linux x86 snapshot testing. >> > > > > > I can see some old discussion in the mailing list about >> > that >> > > > but >> > > > > I >> > > > > > am not sure the expected status here. >> > > > > > They should be intemittent errors since the integrity >> > testing >> > > > > just >> > > > > > run well mostly. >> > > > > > >> > > > > > 3) EUTs: >> > > > > > >> > > > > > 3.1) Linux x86: 99.36% >> > > > > > A recorded JIRA for this suite is HARMONY-2914 which >> wastes >> > > > file >> > > > > > handlers and makes system unstable. >> > > > > > >> > > > > > 4) Functional: >> > > > > > 4.1) Old regressions on both platforms: >> > > > > > api.java.text.MessageFormat (HARMONY-5430) >> > > > > > api.java.util.jar.Manifest (HARMONY-5473) >> > > > > > api.java.beans.beancontext.BeanContextTest (also in M6, >> > > > recorded >> > > > > > as regression caused by changes in locale data) >> > > > > > api.java.beans.persistence.EncoderTest (also in M6, >> > recorded >> > > > as >> > > > > > regression in the beans module) >> > > > > > api.java.beans.persistence.EncoderDecoderTest (also in >> M6, >> > > > > > regression in the beans module) >> > > > > > reg.vm.btest5625 (also in M6, recorded as intermittent >> and >> > > not >> > > > > > reproducible manually) >> > > > > > >> > > > > > 4.2) Old regressions on 1 platform >> > > > > > api.java.rmi.basicexception (ERROR in Linux x86, >> > > HARMONY-5823) >> > > > > > api.java.rmi.basicregistry.RemoteServerTest (ERROR in >> Linux >> > > > x86, >> > > > > > HARMONY-5823) >> > > > > > jpda.jdwp.scenario.ST07.ST07Test (ERROR in windows >> x86_64, >> > in >> > > > M6 >> > > > > it >> > > > > > is recorded as regression since M4) >> > > > > > java.math.F_BigIntegerMatrixMultiplyTest_01 (ERROR on >> Linux >> > > > x86, >> > > > > > recorded as Timeout, not reproducible >> > > > > > in M6) >> > > > > > reg.vm.btest5717 (ERROR on Windows X86_64, recorded as >> > > > "timeout, >> > > > > > the test is too heavy" in M6) >> > > > > > jit.HLO.inline.ControlFlow.IfElse.IfElse1.IfElseTest1 >> > (FAILED >> > > > in >> > > > > > windows x86_64, recorded as "looks like >> > > > > > issue in test" in M6) >> > > > > > jit.HLO.devirt.Runtime.RuntimeExtend1 (FAILED on windows >> > > > x86_64, >> > > > > in >> > > > > > M6 it is recorded as not regression and started to fail on M5) >> > > > > > reg.vm.btest6353.Btest6353 (ERROR on Windows x86_64, >> > recorded >> > > > > also >> > > > > > failed on M3 and might be issue with the test) >> > > > > > >> > > > > > 4.3) New regressions on 1 platform (looks intermittent) >> > > > > > reg.jit.btest8029.Btest8029 (FAILED in Linux x86) >> > > > > > func.reg.jit.btest5710.Btest5710 (FAILED in Linux x86) >> > > > > > >> api.java.security.cert.F_CertPathTest_06.F_CertPathTest_06 >> > > > (ERROR >> > > > > > in Linux x86) >> > > > > > >> api.java.security.cert.F_CertPathTest_05.F_CertPathTest_05 >> > > > (ERROR >> > > > > > in Linux x86) >> > > > > > >> > > > > > 5) JDKTools Tests: >> > > > > > Several timeouts are observed in Linux x86 snapshot running. >> > They >> > > > are: >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch001 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch002 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch003 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch004 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.Events.CombinedEventsTest.testCombinedEvents_04 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.MultiSession.AttachConnectorTest.testAttachConnector001 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.MultiSession.MethodEntryExitTest.testMethodEvent001 >> > > > > > >> > > > > > >> > org.apache.harmony.jpda.tests.jdwp.MultiSession.ResumeTest.testResume >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.MultiSession.ThreadEndTest.testThreadEnd001 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.harmony.jpda.tests.jdwp.MultiSession.ThreadStartTest.testThreadStart001 >> > > > > > The Linux-only timeouts are also observed in the integrity >> > testing >> > > > > > results. >> > > > > > JIRA HARMONY-5833 has been filed for one of them. >> > > > > > >> > > > > > 6) JettyScenario: >> > > > > > The Linux x86 running failed because of the unresolved >> > > HARMONY-5219. >> > > > > > >> > > > > > 7) Reliability: >> > > > > > Several failures are observed in windows x86_64 running. >> > > > > > 7.1) Old regressions >> > > > > > api.net.DatagramTest (HARMONY-5531) >> > > > > > api.text.DecimalFormat_Locales - (in M6 it is recorded as >> > > also >> > > > > > intermittent in M5) >> > > > > > >> > > > > > 7.2) New/intemittent regressions >> > > > > > api.kernel.thread.ThreadLocalTest.ThreadLocalTest >> > > > > > api.kernel.exec.RunExec >> > > > > > >> > > > > > 8) Stress >> > > > > > Different test cases failed on different platforms. >> > > > > > 8.1) Timeouts on Linux x86. >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD003.ThreadTest003 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD007.ThreadTest007 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD009.ThreadTest009 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD011.ThreadTest011 >> > > > > > >> > > > > > 8.2) Failed cases on Windows x86_64 with unknown reasons. >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stress.org.apache.harmony.test.stress.classloader.MixThreads.TreeClasses.testTreeClasses2 >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stress.org.apache.harmony.test.stress.classloader.NotSynchThreads.TreeClasses.testTreeClasses >> > > > > > >> > > > > > I have not found records for them. >> > > > > > >> > > > > > 9) Strut_test >> > > > > > Broken with same error report as M6. >> > > > > > >> > > > > > 10) Eclipse Hello World Application. >> > > > > > Although the testing framework just reports EUT-API status as >> > > > > "PASSED". >> > > > > > A fresh workspace running just failed in configuration stage ( >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://mail-archives.apache.org/mod_mbox/harmony-dev/200805.mbox/[EMAIL >> PROTECTED] >> > > > > > ) >> > > > > > since SVN commit 641928 (which committed patch for HARMONY-4569). >> > > > > > >> > > > > > thanks, >> > > > > > chunrong >> > > > > > >> > > > > >> > > > >> > > >> > >> >
