1) GemfireDataCommandsDUnitTest failed because a TCP port that was
determined to be available was no longer available when the code finally
attempted to use it, resulting in a "java.net.BindException: Address
already in use"
Every test that uses getRandomAvailableTCPPorts is prone to hit this
occasionally. We would have to either:
a) categorize all such tests as Flaky or
b) change the Geode APIs (gemfire.properties) to use "0" to mean that the
product will let the system pick up an ephemeral port during bind instead
of interpreting it as disabling the service that "0" is being specified for
(this would break backwards compatibility for any User using "0" to disable
anything)
final int[] ports = AvailablePortHelper.getRandomAvailableTCPPorts(2);
jmxPort = ports[0];
httpPort = ports[1];
localProps.setProperty(JMX_MANAGER, "true");
localProps.setProperty(JMX_MANAGER_START, "true");
localProps.setProperty(JMX_MANAGER_BIND_ADDRESS,
String.valueOf(jmxHost));
localProps.setProperty(JMX_MANAGER_PORT, String.valueOf(jmxPort));
localProps.setProperty(HTTP_SERVICE_PORT, String.valueOf(httpPort));
getSystem(localProps);
verifyManagementServiceStarted(getCache());
2) ConsoleDistributionManagerDUnitTest failed because the
InternalDistributedSystem connection shut down for an unknown reason during
the test (just as Bruce already mentioned)
ConsoleDistributionManagerDUnitTest is testing the old deprecated Admin
API. We could create a new category such as "DeprecatedTest" to temporarily
exclude such tests from the standard test targets. Ultimately this test and
the Admin API should be removed which requires some investment of time.
Unfortunately, the team currently working on GEODE-17 Integrated Security
doesn't have any time (yet) to dedicate to removal the Admin API and its
tests.
I don't think release/1.0.0-incubating.M3 is any less stable than develop.
I believe these tests could potentially hit the same failures on develop,
though I really don't know what's happening in
ConsoleDistributionManagerDUnitTest.
Is the system that's running the build for release/1.0.0-incubating.M3
different in any way from the infrastructure running the nightly build for
Geode?
-Kirk
On Tue, Jul 19, 2016 at 10:49 AM, Bruce Schuchardt <bschucha...@pivotal.io>
wrote:
> The ConsoleDistributionManagerDUnitTest failure isn't a PDX problem. The
> admin distributed system is being shut down for some reason before the test
> attempts to send an admin request message. Build-artifacts don't have any
> useful information on why that is happening.
>
> Unfortunately the test passes repeatedly when run on its own.
>
>
>
> Le 7/17/2016 à 8:56 PM, Apache Jenkins Server a écrit :
>
>> See <https://builds.apache.org/job/Geode-release/14/>
>>
>> ------------------------------------------
>> [...truncated 697 lines...]
>> java.lang.AssertionError: Suspicious strings were written to the log
>> during this run.
>> Fix the strings or use IgnoredException.addIgnoredException to
>> ignore.
>>
>> -----------------------------------------------------------------------
>> Found suspect string in log4j at line 257
>>
>> [error 2016/07/17 21:46:41.931 UTC <RMI TCP
>> Connection(1)-67.195.81.144> tid=0x1b] Jmx manager could not be started
>> because java.rmi.server.ExportException: Port already in use: 24744; nested
>> exception is:
>> java.net.BindException: Failed to create server socket on
>> asf900.gq1.ygridcore.net/67.195.81.144[24,744]
>> <http://asf900.gq1.ygridcore.net/67.195.81.144%5B24,744%5D>
>> com.gemstone.gemfire.management.ManagementException:
>> java.rmi.server.ExportException: Port already in use: 24744; nested
>> exception is:
>> java.net.BindException: Failed to create server socket on
>> asf900.gq1.ygridcore.net/67.195.81.144[24,744]
>> <http://asf900.gq1.ygridcore.net/67.195.81.144%5B24,744%5D>
>> at
>> com.gemstone.gemfire.management.internal.ManagementAgent.startAgent(ManagementAgent.java:147)
>> at
>> com.gemstone.gemfire.management.internal.SystemManagementService.startManager(SystemManagementService.java:479)
>> at
>> com.gemstone.gemfire.management.internal.beans.ManagementAdapter.handleCacheCreation(ManagementAdapter.java:197)
>> at
>> com.gemstone.gemfire.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:119)
>> at
>> com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2085)
>> at
>> com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:478)
>> at
>> com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1054)
>> at
>> com.gemstone.gemfire.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:707)
>> at
>> com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:695)
>> at
>> com.gemstone.gemfire.cache.CacheFactory.create(CacheFactory.java:181)
>> at
>> com.gemstone.gemfire.cache.CacheFactory.create(CacheFactory.java:172)
>> at
>> com.gemstone.gemfire.test.dunit.cache.internal.JUnit4CacheTestCase.createCache(JUnit4CacheTestCase.java:120)
>> at
>> com.gemstone.gemfire.test.dunit.cache.internal.JUnit4CacheTestCase.getCache(JUnit4CacheTestCase.java:263)
>> at
>> com.gemstone.gemfire.test.dunit.cache.internal.JUnit4CacheTestCase.getCache(JUnit4CacheTestCase.java:242)
>> at
>> com.gemstone.gemfire.test.dunit.cache.internal.JUnit4CacheTestCase.getCache(JUnit4CacheTestCase.java:234)
>> at
>> com.gemstone.gemfire.management.internal.cli.commands.CliCommandTestBase.lambda$setUpJMXManagerOnVM$d0384528$1(CliCommandTestBase.java:154)
>> at
>> com.gemstone.gemfire.test.dunit.NamedCallable.call(NamedCallable.java:33)
>> at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at hydra.MethExecutor.executeObject(MethExecutor.java:268)
>> at
>> com.gemstone.gemfire.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:82)
>> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
>> at sun.rmi.transport.Transport$1.run(Transport.java:200)
>> at sun.rmi.transport.Transport$1.run(Transport.java:197)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.rmi.server.ExportException: Port already in use:
>> 24744; nested exception is:
>> java.net.BindException: Failed to create server socket on
>> asf900.gq1.ygridcore.net/67.195.81.144[24,744]
>> <http://asf900.gq1.ygridcore.net/67.195.81.144%5B24,744%5D>
>> at
>> sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:341)
>> at
>> sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:249)
>> at
>> sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
>> at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
>> at
>> sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:208)
>> at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:152)
>> at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:112)
>> at
>> java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239)
>> at
>> com.gemstone.gemfire.management.internal.ManagementAgent.configureAndStart(ManagementAgent.java:389)
>> at
>> com.gemstone.gemfire.management.internal.ManagementAgent.startAgent(ManagementAgent.java:145)
>> ... 37 more
>> Caused by: java.net.BindException: Failed to create server socket
>> on asf900.gq1.ygridcore.net/67.195.81.144[24,744]
>> <http://asf900.gq1.ygridcore.net/67.195.81.144%5B24,744%5D>
>> at
>> com.gemstone.gemfire.internal.SocketCreator.createServerSocket(SocketCreator.java:814)
>> at
>> com.gemstone.gemfire.internal.SocketCreator.createServerSocket(SocketCreator.java:768)
>> at
>> com.gemstone.gemfire.management.internal.ManagementAgent$GemFireRMIServerSocketFactory.createServerSocket(ManagementAgent.java:545)
>> at
>> sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
>> at
>> sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:330)
>> ... 46 more
>> Caused by: java.net.BindException: Address already in use
>> at java.net.PlainSocketImpl.socketBind(Native Method)
>> at java.net
>> .AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
>> at java.net.ServerSocket.bind(ServerSocket.java:375)
>> at
>> com.gemstone.gemfire.internal.SocketCreator.createServerSocket(SocketCreator.java:811)
>> ... 50 more
>>
>>
>> -----------------------------------------------------------------------
>> Found suspect string in log4j at line 326
>>
>> [error 2016/07/17 21:46:41.938 UTC <RMI TCP
>> Connection(1)-67.195.81.144> tid=0x1b]
>> com.gemstone.gemfire.management.ManagementException:
>> java.rmi.server.ExportException: Port already in use: 24744; nested
>> exception is:
>> java.net.BindException: Failed to create server socket on
>> asf900.gq1.ygridcore.net/67.195.81.144[24,744]
>> <http://asf900.gq1.ygridcore.net/67.195.81.144%5B24,744%5D>
>>
>> 7378 tests completed, 2 failed, 555 skipped
>> :geode-core:distributedTest FAILED
>> :geode-core:flakyTest
>> :geode-core:integrationTest
>> :geode-cq:assemble
>> :geode-cq:compileTestJavaNote: Some input files use or override a
>> deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> Note: Some input files use unchecked or unsafe operations.
>> Note: Recompile with -Xlint:unchecked for details.
>>
>> :geode-cq:processTestResources
>> :geode-cq:testClasses
>> :geode-cq:checkMissedTests
>> :geode-cq:test
>> :geode-cq:check
>> :geode-cq:build
>> :geode-cq:distributedTest
>> :geode-cq:flakyTest
>> :geode-cq:integrationTest
>> :geode-json:assemble
>> :geode-json:compileTestJava UP-TO-DATE
>> :geode-json:processTestResources UP-TO-DATE
>> :geode-json:testClasses UP-TO-DATE
>> :geode-json:checkMissedTests UP-TO-DATE
>> :geode-json:test UP-TO-DATE
>> :geode-json:check
>> :geode-json:build
>> :geode-json:distributedTest UP-TO-DATE
>> :geode-json:flakyTest UP-TO-DATE
>> :geode-json:integrationTest UP-TO-DATE
>> :geode-junit:javadoc
>> :geode-junit:javadocJar
>> :geode-junit:sourcesJar
>> :geode-junit:signArchives SKIPPED
>> :geode-junit:assemble
>> :geode-junit:compileTestJava
>> :geode-junit:processTestResources UP-TO-DATE
>> :geode-junit:testClasses
>> :geode-junit:checkMissedTests
>> :geode-junit:test
>> :geode-junit:check
>> :geode-junit:build
>> :geode-junit:distributedTest
>> :geode-junit:flakyTest
>> :geode-junit:integrationTest
>> :geode-lucene:assemble
>> :geode-lucene:compileTestJavaNote: Some input files use or override a
>> deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> Note: Some input files use unchecked or unsafe operations.
>> Note: Recompile with -Xlint:unchecked for details.
>>
>> :geode-lucene:processTestResources
>> :geode-lucene:testClasses
>> :geode-lucene:checkMissedTests
>> :geode-lucene:test
>> :geode-lucene:check
>> :geode-lucene:build
>> :geode-lucene:distributedTest
>> :geode-lucene:flakyTest
>> :geode-lucene:integrationTest
>> :geode-pulse:assemble
>> :geode-pulse:compileTestJavaNote: <
>> https://builds.apache.org/job/Geode-release/ws/geode-pulse/src/test/java/com/vmware/gemfire/tools/pulse/tests/PulseAbstractTest.java>
>> uses or overrides a deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> Note: Some input files use unchecked or unsafe operations.
>> Note: Recompile with -Xlint:unchecked for details.
>>
>> :geode-pulse:processTestResources
>> :geode-pulse:testClasses
>> :geode-pulse:checkMissedTests
>> :geode-pulse:test
>> :geode-pulse:check
>> :geode-pulse:build
>> :geode-pulse:distributedTest
>> :geode-pulse:flakyTest
>> :geode-pulse:integrationTest
>> :geode-rebalancer:jar
>> :geode-rebalancer:javadoc
>> :geode-rebalancer:javadocJar
>> :geode-rebalancer:sourcesJar
>> :geode-rebalancer:signArchives SKIPPED
>> :geode-rebalancer:assemble
>> :geode-rebalancer:compileTestJava
>> :geode-rebalancer:processTestResources UP-TO-DATE
>> :geode-rebalancer:testClasses
>> :geode-rebalancer:checkMissedTests
>> :geode-rebalancer:test
>> :geode-rebalancer:check
>> :geode-rebalancer:build
>> :geode-rebalancer:distributedTest
>> :geode-rebalancer:flakyTest
>> :geode-rebalancer:integrationTest
>> :geode-wan:assemble
>> :geode-wan:compileTestJavaNote: Some input files use or override a
>> deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> Note: Some input files use unchecked or unsafe operations.
>> Note: Recompile with -Xlint:unchecked for details.
>>
>> :geode-wan:processTestResources
>> :geode-wan:testClasses
>> :geode-wan:checkMissedTests
>> :geode-wan:test
>> :geode-wan:check
>> :geode-wan:build
>> :geode-wan:distributedTest
>> :geode-wan:flakyTest
>> :geode-wan:integrationTest
>> :geode-web:assemble
>> :geode-web:compileTestJavaNote: <
>> https://builds.apache.org/job/Geode-release/ws/geode-web/src/test/java/com/gemstone/gemfire/management/internal/cli/commands/DataCommandsOverHttpDistributedTest.java>
>> uses or overrides a deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> Note: Some input files use unchecked or unsafe operations.
>> Note: Recompile with -Xlint:unchecked for details.
>>
>> :geode-web:processTestResources UP-TO-DATE
>> :geode-web:testClasses
>> :geode-web:checkMissedTests
>> :geode-web:test
>> :geode-web:check
>> :geode-web:build
>> :geode-web:distributedTest
>> :geode-web:flakyTest
>> :geode-web:integrationTest
>> :geode-web-api:assemble
>> :geode-web-api:compileTestJava UP-TO-DATE
>> :geode-web-api:processTestResources UP-TO-DATE
>> :geode-web-api:testClasses UP-TO-DATE
>> :geode-web-api:checkMissedTests UP-TO-DATE
>> :geode-web-api:test UP-TO-DATE
>> :geode-web-api:check
>> :geode-web-api:build
>> :geode-web-api:distributedTest UP-TO-DATE
>> :geode-web-api:flakyTest UP-TO-DATE
>> :geode-web-api:integrationTest UP-TO-DATE
>> :combineReports
>> All test reports at <
>> https://builds.apache.org/job/Geode-release/ws/build/reports/combined>
>> :extensions/geode-modules:precheckin
>> :extensions/geode-modules-assembly:precheckin
>> :extensions/geode-modules-hibernate:precheckin
>> :extensions/geode-modules-session:precheckin
>> :extensions/geode-modules-session-internal:precheckin
>> :extensions/geode-modules-tomcat7:precheckin
>> :geode-assembly:precheckin
>> :geode-common:precheckin
>> :geode-cq:precheckin
>> :geode-json:precheckin
>> :geode-junit:precheckin
>> :geode-lucene:precheckin
>> :geode-pulse:precheckin
>> :geode-rebalancer:precheckin
>> :geode-wan:precheckin
>> :geode-web:precheckin
>> :geode-web-api:precheckin
>>
>> FAILURE: Build failed with an exception.
>>
>> * What went wrong:
>> Execution failed for task ':geode-core:distributedTest'.
>>
>>> There were failing tests. See the report at: file://<
>>> https://builds.apache.org/job/Geode-release/ws/geode-core/build/reports/distributedTest/index.html
>>> >
>>>
>> * Try:
>> Run with --stacktrace option to get the stack trace. Run with --info or
>> --debug option to get more log output.
>>
>> BUILD FAILED
>>
>> Total time: 11 hrs 26 mins 8.356 secs
>> Build step 'Invoke Gradle script' changed build result to FAILURE
>> Build step 'Invoke Gradle script' marked build as failure
>> Archiving artifacts
>> Compressed 250.96 MB of artifacts by 13.1% relative to #7
>> Recording test results
>>
>
>
