#577 hit this same OOME failure. However, #578 did not. It failed with a BindException in DiskRegionJUnitTest.
-Kirk On Thu, Sep 1, 2016 at 1:56 PM, Dan Smith <dsm...@pivotal.io> wrote: > I think we've seen something like this before when tests are logging too > much. Jenkins is trying to parse the test result xml files, which contain > all of the stdout from the test. > > -Dan > > On Thu, Sep 1, 2016 at 1:41 PM, Kirk Lund <kl...@pivotal.io> wrote: > > > We're getting OOME failures in Hudson fairly frequently in the nightly > > build. Any ideas what the issue is? Do we need to change the job > > configuration or exclude more build machines? > > > > -Kirk > > > > ---------- Forwarded message ---------- > > From: Apache Jenkins Server <jenk...@builds.apache.org> > > Date: Thu, Sep 1, 2016 at 1:23 PM > > Subject: Build failed in Jenkins: Geode-nightly #579 > > To: dev@geode.incubator.apache.org, hkhame...@pivotal.io, > gz...@pivotal.io > > , > > upthewatersp...@apache.org, sbawas...@pivotal.io, bogle...@pivotal.io, > > aba...@pivotal.io > > > > > > See <https://builds.apache.org/job/Geode-nightly/579/changes> > > > > Changes: > > > > [abaker] GEODE-1809: Improve error handling > > > > [gzhou] GEODE-11: create index repository using raw Lucene directory. > > > > [gzhou] GEODE-11: need to fix the junit test case when adding FSDirectory > > > > ------------------------------------------ > > [...truncated 486 lines...] > > [fatal 2016/09/01 10:44:41.170 UTC <JMX client heartbeat 2> > tid=0x1e1] > > (tid=481 msgId=19) No longer connected to localhost[27402]. > > > > 7606 tests completed, 1 failed, 588 skipped > > :geode-core:distributedTest FAILED > > :geode-core:flakyTest > > :geode-core:integrationTest > > :geode-cq:assemble > > :geode-cq:compileTestJavaNote: Some input files use or override a > > deprecated API. > > Note: Recompile with -Xlint:deprecation for details. > > Note: Some input files use unchecked or unsafe operations. > > Note: Recompile with -Xlint:unchecked for details. > > > > :geode-cq:processTestResources > > :geode-cq:testClasses > > :geode-cq:checkMissedTests > > :geode-cq:test > > :geode-cq:check > > :geode-cq:build > > :geode-cq:distributedTest > > :geode-cq:flakyTest > > :geode-cq:integrationTest > > :geode-json:assemble > > :geode-json:compileTestJava UP-TO-DATE > > :geode-json:processTestResources UP-TO-DATE > > :geode-json:testClasses UP-TO-DATE > > :geode-json:checkMissedTests UP-TO-DATE > > :geode-json:test UP-TO-DATE > > :geode-json:check > > :geode-json:build > > :geode-json:distributedTest UP-TO-DATE > > :geode-json:flakyTest UP-TO-DATE > > :geode-json:integrationTest UP-TO-DATE > > :geode-junit:javadoc > > :geode-junit:javadocJar > > :geode-junit:sourcesJar > > :geode-junit:signArchives SKIPPED > > :geode-junit:assemble > > :geode-junit:compileTestJava > > :geode-junit:processTestResources UP-TO-DATE > > :geode-junit:testClasses > > :geode-junit:checkMissedTests > > :geode-junit:test > > :geode-junit:check > > :geode-junit:build > > :geode-junit:distributedTest > > :geode-junit:flakyTest > > :geode-junit:integrationTest > > :geode-lucene:assemble > > :geode-lucene:compileTestJavaNote: Some input files use or override a > > deprecated API. > > Note: Recompile with -Xlint:deprecation for details. > > Note: Some input files use unchecked or unsafe operations. > > Note: Recompile with -Xlint:unchecked for details. > > > > :geode-lucene:processTestResources > > :geode-lucene:testClasses > > :geode-lucene:checkMissedTests > > :geode-lucene:test > > :geode-lucene:check > > :geode-lucene:build > > :geode-lucene:distributedTest > > :geode-lucene:flakyTest > > :geode-lucene:integrationTest > > :geode-pulse:assemble > > :geode-pulse:compileTestJavaNote: <https://builds.apache.org/ > > job/Geode-nightly/ws/geode-pulse/src/test/java/com/ > > vmware/gemfire/tools/pulse/tests/PulseAbstractTest.java> uses or > overrides > > a deprecated API. > > Note: Recompile with -Xlint:deprecation for details. > > Note: Some input files use unchecked or unsafe operations. > > Note: Recompile with -Xlint:unchecked for details. > > > > :geode-pulse:processTestResources > > :geode-pulse:testClasses > > :geode-pulse:checkMissedTests > > :geode-pulse:test > > :geode-pulse:check > > :geode-pulse:build > > :geode-pulse:distributedTest > > :geode-pulse:flakyTest > > :geode-pulse:integrationTest > > :geode-rebalancer:jar > > :geode-rebalancer:javadoc > > :geode-rebalancer:javadocJar > > :geode-rebalancer:sourcesJar > > :geode-rebalancer:signArchives SKIPPED > > :geode-rebalancer:assemble > > :geode-rebalancer:compileTestJava > > :geode-rebalancer:processTestResources UP-TO-DATE > > :geode-rebalancer:testClasses > > :geode-rebalancer:checkMissedTests > > :geode-rebalancer:test > > :geode-rebalancer:check > > :geode-rebalancer:build > > :geode-rebalancer:distributedTest > > :geode-rebalancer:flakyTest > > :geode-rebalancer:integrationTest > > :geode-wan:assemble > > :geode-wan:compileTestJavaNote: Some input files use or override a > > deprecated API. > > Note: Recompile with -Xlint:deprecation for details. > > Note: Some input files use unchecked or unsafe operations. > > Note: Recompile with -Xlint:unchecked for details. > > > > :geode-wan:processTestResources > > :geode-wan:testClasses > > :geode-wan:checkMissedTests > > :geode-wan:test > > :geode-wan:check > > :geode-wan:build > > :geode-wan:distributedTest > > :geode-wan:flakyTest > > :geode-wan:integrationTest > > :geode-web:assemble > > :geode-web:compileTestJavaNote: <https://builds.apache.org/ > > job/Geode-nightly/ws/geode-web/src/test/java/com/ > > gemstone/gemfire/management/internal/cli/commands/ > > DataCommandsOverHttpDistributedTest.java> uses or overrides a deprecated > > API. > > Note: Recompile with -Xlint:deprecation for details. > > Note: Some input files use unchecked or unsafe operations. > > Note: Recompile with -Xlint:unchecked for details. > > > > :geode-web:processTestResources UP-TO-DATE > > :geode-web:testClasses > > :geode-web:checkMissedTests > > :geode-web:test > > :geode-web:check > > :geode-web:build > > :geode-web:distributedTest > > :geode-web:flakyTest > > :geode-web:integrationTest > > :geode-web-api:assemble > > :geode-web-api:compileTestJava UP-TO-DATE > > :geode-web-api:processTestResources UP-TO-DATE > > :geode-web-api:testClasses UP-TO-DATE > > :geode-web-api:checkMissedTests UP-TO-DATE > > :geode-web-api:test UP-TO-DATE > > :geode-web-api:check > > :geode-web-api:build > > :geode-web-api:distributedTest UP-TO-DATE > > :geode-web-api:flakyTest UP-TO-DATE > > :geode-web-api:integrationTest UP-TO-DATE > > :combineReports > > All test reports at <https://builds.apache.org/ > job/Geode-nightly/ws/build/ > > reports/combined> > > :extensions/geode-modules:precheckin > > :extensions/geode-modules-assembly:precheckin > > :extensions/geode-modules-hibernate:precheckin > > :extensions/geode-modules-session:precheckin > > :extensions/geode-modules-session-internal:precheckin > > :extensions/geode-modules-tomcat7:precheckin > > :extensions/geode-modules-tomcat8:precheckin > > :geode-assembly:precheckin > > :geode-common:precheckin > > :geode-cq:precheckin > > :geode-json:precheckin > > :geode-junit:precheckin > > :geode-lucene:precheckin > > :geode-pulse:precheckin > > :geode-rebalancer:precheckin > > :geode-wan:precheckin > > :geode-web:precheckin > > :geode-web-api:precheckin > > :extensions/geode-modules:uploadArchives > > :extensions/geode-modules-assembly:uploadArchives SKIPPED > > :extensions/geode-modules-hibernate:uploadArchives > > :extensions/geode-modules-session:uploadArchives > > :extensions/geode-modules-session-internal:uploadArchives > > :extensions/geode-modules-tomcat7:uploadArchives > > :extensions/geode-modules-tomcat8:uploadArchives > > :geode-assembly:uploadArchives SKIPPED > > :geode-common:uploadArchives > > :geode-core:uploadArchives > > :geode-cq:uploadArchives > > :geode-json:uploadArchives > > :geode-junit:uploadArchives > > :geode-lucene:uploadArchives > > :geode-pulse:uploadArchives > > :geode-rebalancer:uploadArchives > > :geode-wan:uploadArchives > > :geode-web:uploadArchives > > :geode-web-api:uploadArchives > > > > FAILURE: Build failed with an exception. > > > > * What went wrong: > > Execution failed for task ':geode-core:distributedTest'. > > > There were failing tests. See the report at: file://< > > https://builds.apache.org/job/Geode-nightly/ws/geode-core/build/reports/ > > distributedTest/index.html> > > > > * Try: > > Run with --stacktrace option to get the stack trace. Run with --info or > > --debug option to get more log output. > > > > BUILD FAILED > > > > Total time: 11 hrs 10 mins 1.029 secs > > Build step 'Invoke Gradle script' changed build result to FAILURE > > Build step 'Invoke Gradle script' marked build as failure > > Archiving artifacts > > Compressed 378.83 MB of artifacts by 16.1% relative to #573 > > Recording test results > > ERROR: Step ?Publish JUnit test result report? aborted due to exception: > > java.io.IOException: remote file operation failed: < > > https://builds.apache.org/job/Geode-nightly/ws/> at > > hudson.remoting.Channel@43724f7c:H4: > > java.io.IOException: Remote call on H4 failed > > at hudson.FilePath.act(FilePath.java:1014) > > at hudson.FilePath.act(FilePath.java:996) > > at hudson.tasks.junit.JUnitParser.parseResult( > > JUnitParser.java:103) > > at hudson.tasks.junit.JUnitResultArchiver.parse( > > JUnitResultArchiver.java:128) > > at hudson.tasks.junit.JUnitResultArchiver.perform( > > JUnitResultArchiver.java:149) > > at hudson.tasks.BuildStepCompatibilityLayer.perform( > > BuildStepCompatibilityLayer.java:78) > > at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor. > > java:20) > > at hudson.model.AbstractBuild$AbstractBuildExecution. > > perform(AbstractBuild.java:779) > > at hudson.model.AbstractBuild$AbstractBuildExecution. > > performAllBuildSteps(AbstractBuild.java:720) > > at hudson.model.Build$BuildExecution.post2(Build.java:185) > > at hudson.model.AbstractBuild$AbstractBuildExecution.post( > > AbstractBuild.java:665) > > at hudson.model.Run.execute(Run.java:1766) > > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > > at hudson.model.ResourceController.execute( > > ResourceController.java:98) > > at hudson.model.Executor.run(Executor.java:410) > > Caused by: java.io.IOException: Remote call on H4 failed > > at hudson.remoting.Channel.call(Channel.java:789) > > at hudson.FilePath.act(FilePath.java:1007) > > ... 14 more > > Caused by: java.lang.OutOfMemoryError: Java heap space > > at com.sun.org.apache.xerces.internal.util.XMLStringBuffer. > > append(XMLStringBuffer.java:208) > > at com.sun.org.apache.xerces.internal.impl. > > XMLEntityScanner.scanData(XMLEntityScanner.java:1507) > > at com.sun.org.apache.xerces.internal.impl. > > XMLDocumentFragmentScannerImpl.scanCDATASection( > > XMLDocumentFragmentScannerImpl.java:1648) > > at com.sun.org.apache.xerces.internal.impl. > > XMLDocumentFragmentScannerImpl$FragmentContentDriver.next( > > XMLDocumentFragmentScannerImpl.java:3011) > > at com.sun.org.apache.xerces.internal.impl. > > XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) > > at com.sun.org.apache.xerces.internal.impl. > > XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118) > > at com.sun.org.apache.xerces.internal.impl. > > XMLDocumentFragmentScannerImpl.scanDocument( > XMLDocumentFragmentScannerImpl > > .java:504) > > at com.sun.org.apache.xerces.internal.parsers. > > XML11Configuration.parse(XML11Configuration.java:848) > > at com.sun.org.apache.xerces.internal.parsers. > > XML11Configuration.parse(XML11Configuration.java:777) > > at com.sun.org.apache.xerces.internal.parsers.XMLParser. > > parse(XMLParser.java:141) > > at com.sun.org.apache.xerces.internal.parsers. > > AbstractSAXParser.parse(AbstractSAXParser.java:1213) > > at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$ > > JAXPSAXParser.parse(SAXParserImpl.java:642) > > at org.dom4j.io.SAXReader.read(SAXReader.java:465) > > at org.dom4j.io.SAXReader.read(SAXReader.java:264) > > at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:126) > > at hudson.tasks.junit.TestResult.parse(TestResult.java:301) > > at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult. > > java:244) > > at hudson.tasks.junit.TestResult.parse(TestResult.java:175) > > at hudson.tasks.junit.TestResult.parse(TestResult.java:154) > > at hudson.tasks.junit.TestResult.<init>(TestResult.java:126) > > at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke( > > JUnitParser.java:132) > > at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke( > > JUnitParser.java:107) > > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772) > > at hudson.remoting.UserRequest.perform(UserRequest.java:153) > > at hudson.remoting.UserRequest.perform(UserRequest.java:50) > > at hudson.remoting.Request$2.run(Request.java:332) > > at hudson.remoting.InterceptingExecutorService$1.call( > > InterceptingExecutorService.java:68) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > at ......remote call to H4(Native Method) > > at hudson.remoting.Channel.attachCallSiteStackTrace( > > Channel.java:1416) > > at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) > > at hudson.remoting.Channel.call(Channel.java:781) > > ... 15 more > > >
