Hi Kent, There is no point sending multiple emails for the same subject. It distracts people from the other messages, distributes the conversation and discourages people from helping you.
Please provide more details about your cluster. 1. How many nodes? 2. How many collections? 3. How many shards? 4. What is the replication factor? 5. Are there connection and read timeouts specified on both client and server? If yes, what are the values? 6. How often do you commit? Is there auto-commit or explicit commits from clients? 7. What else is happening on your cluster? What is the write and query load? The important thing is to figure out why these pools are getting exhausted. 10000 max connections and 500 connections per host is already pretty high so there is an underlying cause for such stuck requests. On Wed, Jul 6, 2016 at 4:56 PM, Kent Mu <solr.st...@gmail.com> wrote: > Hi friends! > *solr version: 4.9.0* > > I came across a problem when use solrcloud, it becomes dead lock, we got > the java core log, it looks like the http connection pool is exhausted and > most threads are waiting to get a free connection.. > > I posted the problem in JIRA, the link is > https://issues.apache.org/jira/browse/SOLR-9253 > I have increased http connection defaults for the SolrJ client, and also > configed the connection defaults in solr.xml for all shard servers as > below. > > <shardHandlerFactory name="shardHandlerFactory" > class="HttpShardHandlerFactory"> > <int name="socketTimeout">60000</int> > <int name="connTimeout">30000</int> > <int name="maxConnections">10000</int> > <int name="maxConnectionsPerHost">500</int> > </shardHandlerFactory> > > *besides, we use the singleton pattern to connect solrlcoud.* > > public synchronized static CloudSolrServer getSolrCloudReadServer() { > if (reviewSolrCloudServer == null) { > ModifiableSolrParams params = new ModifiableSolrParams(); > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000); > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500); > HttpClient client = HttpClientUtil.createClient(params); > LBHttpSolrServer lbServer = new LBHttpSolrServer(client); > > > lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut()); > lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut()); > > reviewSolrCloudServer = new > CloudSolrServer(ReviewProperties.getZkHost(),lbServer); > reviewSolrCloudServer.setDefaultCollection("commodityReview"); > > > reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout()); > > > reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout()); > reviewSolrCloudServer.connect(); > } > return reviewSolrCloudServer; > } > > *the java stack as below* > > "httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000 nid=0x20af > waiting on condition [0x00007f79fd490000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000605710068> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133) > at > > org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282) > at > org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) > at > > org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177) > at > > org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170) > at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102) > at > > org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208) > at > > org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195) > at > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422) > at > > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) > at > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > at > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) > at > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) > at > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340) > at > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301) > at > > org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > > please help me, look forward to your reply. Thanks a lot! > > Best Regards! > Kent > -- Regards, Shalin Shekhar Mangar.
