Gus Heck created SOLR-13036: ------------------------------- Summary: JettySolrRunner's port retry is broken Key: SOLR-13036 URL: https://issues.apache.org/jira/browse/SOLR-13036 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: Tests Affects Versions: master (8.0) Reporter: Gus Heck
I was running tests and hit the following stack trace which shows the BindException getting wrapped by an IOException, which then evades our port already bound retry logic in JettySolrRunner#retryOnPortBindFailure() which is catching BindException directly. The result is that we never retry as intended. This observation from reading the code is supported by the fact that the logging statement starting with "Port in use..." did not appear in the log. {code:java} build.out4.txt- [beaster] [18:06:13.618] ERROR 25.9s J15 | TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader <<< build.out4.txt: [beaster] > Throwable #1: java.io.IOException: Failed to bind to /127.0.0.1:36497 build.out4.txt- [beaster] > at __randomizedtesting.SeedInfo.seed([FD0124C33BA78623:83EA04D3F8C08919]:0) build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346) build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308) build.out4.txt- [beaster] > at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) build.out4.txt- [beaster] > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) build.out4.txt- [beaster] > at org.eclipse.jetty.server.Server.doStart(Server.java:394) build.out4.txt- [beaster] > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:510) build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:449) build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:417) build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.addDocWhenOtherReplicasAreNetworkPartitioned(TestCloudConsistency.java:225) build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:136) build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:98) build.out4.txt- [beaster] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) build.out4.txt- [beaster] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) build.out4.txt- [beaster] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) build.out4.txt- [beaster] > at java.lang.reflect.Method.invoke(Method.java:498) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) build.out4.txt- [beaster] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) build.out4.txt- [beaster] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) build.out4.txt- [beaster] > at java.lang.Thread.run(Thread.java:748) build.out4.txt- [beaster] > Caused by: java.net.BindException: Address already in use build.out4.txt- [beaster] > at sun.nio.ch.Net.bind0(Native Method) build.out4.txt- [beaster] > at sun.nio.ch.Net.bind(Net.java:433) build.out4.txt- [beaster] > at sun.nio.ch.Net.bind(Net.java:425) build.out4.txt- [beaster] > at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) build.out4.txt- [beaster] > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342) build.out4.txt- [beaster] > ... 51 more {code} SOLR-7339 first noticed this but seems to have not tracked it down because other bigger problems were extant. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org