Hi, I think the issue was not in zk client timeout, but POST request size. When I increased the value for Request.maxFormContentSize in jetty.xml I don't see this issue any more.
Regards. On 3 February 2013 01:56, Mark Miller <markrmil...@gmail.com> wrote: > Do you see anything about session expiration in the logs? That is the > likely culprit for something like this. You may need to raise the timeout: > http://wiki.apache.org/solr/SolrCloud#FAQ > > If you see no session timeouts, I don't have a guess yet. > > - Mark > > On Feb 2, 2013, at 7:35 PM, Marcin Rzewucki <mrzewu...@gmail.com> wrote: > > > I'm experiencing same problem in Solr4.1 during bulk loading. After 50 > > minutes of indexing the following error starts to occur: > > > > INFO: [core] webapp=/solr path=/update params={} {} 0 4 > > Feb 02, 2013 11:36:15 PM org.apache.solr.common.SolrException log > > SEVERE: org.apache.solr.common.SolrException: ClusterState says we are > the > > leader, but locally we don't think so > > at > > > org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:295) > > at > > > org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:230) > > at > > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:343) > > at > > > org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) > > at > > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:387) > > at > > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:112) > > at > > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:96) > > at > > org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:60) > > at > > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) > > at > > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) > > at > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > > at > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > > at > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > > at org.eclipse.jetty.server.Server.handle(Server.java:365) > > at > > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) > > at > > > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) > > at > > > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937) > > at > > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998) > > at > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856) > > at > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) > > at > > > org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) > > at > > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > > at java.lang.Thread.run(Unknown Source) > > Feb 02, 2013 11:36:15 PM org.apache.solr.common.SolrException log > > Feb 02, 2013 11:36:31 PM org.apache.solr.cloud.ShardLeaderElectionContext > > waitForReplicasToComeUp > > INFO: Waiting until we see more replicas up: total=2 found=1 > timeoutin=50699 > > > > Then leader tries to sync with replica and after it finishes I can > continue > > loading. > > None of SolrCloud nodes was restarted during that time. I don't remember > > such behaviour in Solr4.0. Could it be related with the number of fields > > indexed during loading ? I have a collection with about 2400 fields. I > > can't reproduce same issue for other collections with much less fields > per > > record. > > Regards. > > > > On 11 December 2012 19:50, Sudhakar Maddineni <maddineni...@gmail.com > >wrote: > > > >> Just an update on this issue: > >> We tried by increasing zookeeper client timeout settings to 30000ms in > >> solr.xml (i think default is 15000ms), and haven't seen any issues from > our > >> tests. > >> <cores ......... zkClientTimeout="30000" > > >> > >> Thanks, Sudhakar. > >> > >> On Fri, Dec 7, 2012 at 4:55 PM, Sudhakar Maddineni > >> <maddineni...@gmail.com>wrote: > >> > >>> We saw this error again today during our load test - basically, > whenever > >>> session is getting expired on the leader node, we are seeing the > >>> error.After this happens, leader(001) is going into 'recovery' mode and > >> all > >>> the index updates are failing with "503- service unavailable" error > >>> message.After some time(once recovery is successful), roles are swapped > >>> i.e. 001 acting as the replica and 003 as leader. > >>> > >>> Btw, do you know why the connection to zookeeper[solr->zk] getting > >>> interrupted in the middle? > >>> is it because of the load(no of updates) we are putting on the cluster? > >>> > >>> Here is the exception stack trace: > >>> > >>> *Dec* *7*, *2012* *2:28:03* *PM* > >> *org.apache.solr.cloud.Overseer$ClusterStateUpdater* *amILeader* > >>> *WARNING:* > >> *org.apache.zookeeper.KeeperException$SessionExpiredException:* > >> *KeeperErrorCode* *=* *Session* *expired* *for* */overseer_elect/leader* > >>> *at* > >> > *org.apache.zookeeper.KeeperException.create*(*KeeperException.java:118*) > >>> *at* > >> *org.apache.zookeeper.KeeperException.create*(*KeeperException.java:42*) > >>> *at* *org.apache.zookeeper.ZooKeeper.getData*(*ZooKeeper.java:927* > >>> ) > >>> *at* > >> > *org.apache.solr.common.cloud.SolrZkClient$7.execute*(*SolrZkClient.java:244*) > >>> *at* > >> > *org.apache.solr.common.cloud.SolrZkClient$7.execute*(*SolrZkClient.java:241*) > >>> *at* > >> > *org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation*(*ZkCmdExecutor.java:63*) > >>> *at* > >> > *org.apache.solr.common.cloud.SolrZkClient.getData*(*SolrZkClient.java:241*) > >>> *at* > >> > *org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader*(*Overseer.java:195*) > >>> *at* > >> > *org.apache.solr.cloud.Overseer$ClusterStateUpdater.run*(*Overseer.java:119*) > >>> *at* *java.lang.Thread.run*(*Unknown* *Source*) > >>> > >>> Thx,Sudhakar. > >>> > >>> > >>> > >>> On Fri, Dec 7, 2012 at 3:16 PM, Sudhakar Maddineni < > >> maddineni...@gmail.com > >>>> wrote: > >>> > >>>> Erick: > >>>> Not seeing any page caching related issues... > >>>> > >>>> Mark: > >>>> 1.Would this "waiting" on 003(replica) cause any inconsistencies in > >> the > >>>> zookeeper cluster state? I was also looking at the leader(001) logs at > >> that > >>>> time and seeing errors related to "*SEVERE: ClusterState says we are > the > >>>> leader, but locally we don't think so*". > >>>> 2.Also, all of our servers in cluster were gone down when the index > >>>> updates were running in parallel along with this issue.Do you see this > >>>> related to the session expiry on 001? > >>>> > >>>> Here are the logs on 001 > >>>> ========================= > >>>> > >>>> Dec 4, 2012 12:12:29 PM > >>>> org.apache.solr.cloud.Overseer$ClusterStateUpdater amILeader > >>>> WARNING: > >>>> org.apache.zookeeper.KeeperException$SessionExpiredException: > >>>> KeeperErrorCode = Session expired for /overseer_elect/leader > >>>> at > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > >>>> at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > >>>> Dec 4, 2012 12:12:29 PM > >>>> org.apache.solr.cloud.Overseer$ClusterStateUpdater amILeader > >>>> INFO: According to ZK I > >>>> (id=232887758696546307-<001>:8080_solr-n_0000000005) am no longer a > >> leader. > >>>> > >>>> Dec 4, 2012 12:12:29 PM > >> org.apache.solr.cloud.OverseerCollectionProcessor > >>>> run > >>>> WARNING: Overseer cannot talk to ZK > >>>> > >>>> Dec 4, 2012 12:13:00 PM org.apache.solr.common.SolrException log > >>>> SEVERE: There was a problem finding the leader in > >>>> zk:org.apache.solr.common.SolrException: Could not get leader props > >>>> at > >>>> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:709) > >>>> at > >>>> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:673) > >>>> Dec 4, 2012 12:13:32 PM org.apache.solr.common.SolrException log > >>>> SEVERE: There was a problem finding the leader in > >>>> zk:org.apache.solr.common.SolrException: Could not get leader props > >>>> at > >>>> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:709) > >>>> at > >>>> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:673) > >>>> Dec 4, 2012 12:15:17 PM org.apache.solr.common.SolrException log > >>>> SEVERE: There was a problem making a request to the > >>>> leader:org.apache.solr.common.SolrException: I was asked to wait on > >> state > >>>> down for <001>:8080_solr but I still do not see the request state. I > see > >>>> state: active live:true > >>>> at > >>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:401) > >>>> Dec 4, 2012 12:15:50 PM org.apache.solr.common.SolrException log > >>>> SEVERE: There was a problem making a request to the > >>>> leader:org.apache.solr.common.SolrException: I was asked to wait on > >> state > >>>> down for <001>:8080_solr but I still do not see the request state. I > see > >>>> state: active live:true > >>>> at > >>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:401) > >>>> .... > >>>> .... > >>>> Dec 4, 2012 12:19:10 PM org.apache.solr.common.SolrException log > >>>> SEVERE: There was a problem finding the leader in > >>>> zk:org.apache.solr.common.SolrException: Could not get leader props > >>>> at > >>>> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:709) > >>>> .... > >>>> .... > >>>> Dec 4, 2012 12:21:24 PM org.apache.solr.common.SolrException log > >>>> SEVERE: :org.apache.solr.common.SolrException: There was a problem > >>>> finding the leader in zk > >>>> at > >>>> > >> > org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1080) > >>>> at > >>>> > >> > org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:273) > >>>> Dec 4, 2012 12:22:30 PM org.apache.solr.cloud.ZkController getLeader > >>>> SEVERE: Error getting leader from zk > >>>> org.apache.solr.common.SolrException: *There is conflicting > information > >>>> about the leader of shard: shard1 our state says:http:// > >> <001>:8080/solr/core1/ > >>>> but zookeeper says:http://<003>:8080/solr/core1/* > >>>> * at > >> org.apache.solr.cloud.ZkController.getLeader(ZkController.java:647)* > >>>> * at > org.apache.solr.cloud.ZkController.register(ZkController.java:577)* > >>>> Dec 4, 2012 12:22:30 PM > >>>> org.apache.solr.cloud.ShardLeaderElectionContext runLeaderProcess > >>>> INFO: Running the leader process. > >>>> .... > >>>> .... > >>>> > >>>> Thanks for your inputs. > >>>> Sudhakar. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Dec 6, 2012 at 5:35 PM, Mark Miller <markrmil...@gmail.com > >>> wrote: > >>>> > >>>>> Yes - it means that 001 went down (or more likely had it's connection > >> to > >>>>> ZooKeeper interrupted? that's what I mean about a session timeout - > if > >> the > >>>>> solr->zk link is broken for longer than the session timeout that will > >>>>> trigger a leader election and when the connection is reestablished, > the > >>>>> node will have to recover). That waiting should stop as soon as 001 > >> came > >>>>> back up or reconnected to ZooKeeper. > >>>>> > >>>>> In fact, this waiting should not happen in this case - but only on > >>>>> cluster restart. This is a bug that is fixed in 4.1 (hopefully coming > >> very > >>>>> soon!): > >>>>> > >>>>> * SOLR-3940: Rejoining the leader election incorrectly triggers the > >> code > >>>>> path > >>>>> for a fresh cluster start rather than fail over. (Mark Miller) > >>>>> > >>>>> - Mark > >>>>> > >>>>> On Dec 5, 2012, at 9:41 PM, Sudhakar Maddineni < > maddineni...@gmail.com > >>> > >>>>> wrote: > >>>>> > >>>>>> Yep, after restarting, cluster came back to normal state.We will run > >>>>> couple of more tests and see if we could reproduce this issue. > >>>>>> > >>>>>> Btw, I am attaching the server logs before that 'INFO: Waiting until > >>>>> we see more replicas' message.From the logs, we can see that leader > >>>>> election process started on 003 which was the replica for 001 > >>>>> initially.That means leader 001 went down at that time? > >>>>>> > >>>>>> logs on 003: > >>>>>> ======== > >>>>>> 12:11:16 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> runLeaderProcess > >>>>>> INFO: Running the leader process. > >>>>>> 12:11:16 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> shouldIBeLeader > >>>>>> INFO: Checking if I should try and be the leader. > >>>>>> 12:11:16 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> shouldIBeLeader > >>>>>> INFO: My last published State was Active, it's okay to be the > >>>>> leader. > >>>>>> 12:11:16 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> runLeaderProcess > >>>>>> INFO: I may be the new leader - try and sync > >>>>>> 12:11:16 PM org.apache.solr.cloud.RecoveryStrategy close > >>>>>> WARNING: Stopping recovery for > >> zkNodeName=<003>:8080_solr_core > >>>>> core=core1. > >>>>>> 12:11:16 PM org.apache.solr.cloud.SyncStrategy sync > >>>>>> INFO: Sync replicas to http://<003>:8080/solr/core1/ > >>>>>> 12:11:16 PM org.apache.solr.update.PeerSync sync > >>>>>> INFO: PeerSync: core=core1 url=http://<003>:8080/solr START > >>>>> replicas=[<001>:8080/solr/core1/] nUpdates=100 > >>>>>> 12:11:16 PM org.apache.solr.common.cloud.ZkStateReader$3 process > >>>>>> INFO: Updating live nodes -> this message is on 002 > >>>>>> 12:11:46 PM org.apache.solr.update.PeerSync handleResponse > >>>>>> WARNING: PeerSync: core=core1 url=http://<003>:8080/solr > >>>>> exception talking to <001>:8080/solr/core1/, failed > >>>>>> org.apache.solr.client.solrj.SolrServerException: Timeout > >>>>> occured while waiting response from server at: <001>:8080/solr/core1 > >>>>>> at > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409) > >>>>>> at > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) > >>>>>> at > >>>>> > >> > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:166) > >>>>>> at > >>>>> > >> > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:133) > >>>>>> at > >> java.util.concurrent.FutureTask$Sync.innerRun(Unknown > >>>>> Source) > >>>>>> at java.util.concurrent.FutureTask.run(Unknown Source) > >>>>>> at > >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > >>>>>> at > >> java.util.concurrent.FutureTask$Sync.innerRun(Unknown > >>>>> Source) > >>>>>> at java.util.concurrent.FutureTask.run(Unknown Source) > >>>>>> at > >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > >>>>>> at > >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >>>>>> at java.lang.Thread.run(Unknown Source) > >>>>>> Caused by: java.net.SocketTimeoutException: Read timed out > >>>>>> at java.net.SocketInputStream.socketRead0(Native > >> Method) > >>>>>> at java.net.SocketInputStream.read(Unknown Source) > >>>>>> 12:11:46 PM org.apache.solr.update.PeerSync sync > >>>>>> INFO: PeerSync: core=core1 url=http://<003>:8080/solr DONE. > >>>>> sync failed > >>>>>> 12:11:46 PM org.apache.solr.common.SolrException log > >>>>>> SEVERE: Sync Failed > >>>>>> 12:11:46 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> rejoinLeaderElection > >>>>>> INFO: There is a better leader candidate than us - going back > >>>>> into recovery > >>>>>> 12:11:46 PM org.apache.solr.update.DefaultSolrCoreState doRecovery > >>>>>> INFO: Running recovery - first canceling any ongoing recovery > >>>>>> 12:11:46 PM org.apache.solr.cloud.RecoveryStrategy run > >>>>>> INFO: Starting recovery process. core=core1 > >>>>> recoveringAfterStartup=false > >>>>>> 12:11:46 PM org.apache.solr.cloud.RecoveryStrategy doRecovery > >>>>>> INFO: Attempting to PeerSync from <001>:8080/solr/core1/ > >>>>> core=core1 - recoveringAfterStartup=false > >>>>>> 12:11:46 PM org.apache.solr.update.PeerSync sync > >>>>>> INFO: PeerSync: core=core1 url=http://<003>:8080/solr START > >>>>> replicas=[<001>:8080/solr/core1/] nUpdates=100 > >>>>>> 12:11:46 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> runLeaderProcess > >>>>>> INFO: Running the leader process. > >>>>>> 12:11:46 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> waitForReplicasToComeUp > >>>>>> INFO: Waiting until we see more replicas up: total=2 found=1 > >>>>> timeoutin=179999 > >>>>>> 12:11:47 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> waitForReplicasToComeUp > >>>>>> INFO: Waiting until we see more replicas up: total=2 found=1 > >>>>> timeoutin=179495 > >>>>>> 12:11:48 PM org.apache.solr.cloud.ShardLeaderElectionContext > >>>>> waitForReplicasToComeUp > >>>>>> INFO: Waiting until we see more replicas up: total=2 found=1 > >>>>> timeoutin=178985 > >>>>>> .... > >>>>>> .... > >>>>>> > >>>>>> Thanks for your help. > >>>>>> Sudhakar. > >>>>>> > >>>>>> On Wed, Dec 5, 2012 at 6:19 PM, Mark Miller <markrmil...@gmail.com> > >>>>> wrote: > >>>>>> The waiting logging had to happen on restart unless it's some kind > of > >>>>> bug. > >>>>>> > >>>>>> Beyond that, something is off, but I have no clue why - it seems > your > >>>>> clusterstate.json is not up to date at all. > >>>>>> > >>>>>> Have you tried restarting the cluster then? Does that help at all? > >>>>>> > >>>>>> Do you see any exceptions around zookeeper session timeouts? > >>>>>> > >>>>>> - Mark > >>>>>> > >>>>>> On Dec 5, 2012, at 4:57 PM, Sudhakar Maddineni < > >> maddineni...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>>> Hey Mark, > >>>>>>> > >>>>>>> Yes, I am able to access all of the nodes under each shard from > >>>>> solrcloud > >>>>>>> admin UI. > >>>>>>> > >>>>>>> > >>>>>>> - *It kind of looks like the urls solrcloud is using are not > >>>>> accessible. > >>>>>>> When you go to the admin page and the cloud tab, can you access > >>>>> the urls it > >>>>>>> shows for each shard? That is, if you click on of the links or > >>>>> copy and > >>>>>>> paste the address into a web browser, does it work?* > >>>>>>> > >>>>>>> Actually, I got these errors when my document upload task/job was > >>>>> running, > >>>>>>> not during the cluster restart. Also,job ran fine initially for the > >>>>> first > >>>>>>> one hour and started throwing these errors after indexing some > >> docx. > >>>>>>> > >>>>>>> Thx, Sudhakar. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Dec 5, 2012 at 5:38 PM, Mark Miller <markrmil...@gmail.com > >>> > >>>>> wrote: > >>>>>>> > >>>>>>>> It kind of looks like the urls solrcloud is using are not > >>>>> accessible. When > >>>>>>>> you go to the admin page and the cloud tab, can you access the > >> urls > >>>>> it > >>>>>>>> shows for each shard? That is, if you click on of the links or > >> copy > >>>>> and > >>>>>>>> paste the address into a web browser, does it work? > >>>>>>>> > >>>>>>>> You may have to explicitly set the host= in solr.xml if it's not > >>>>> auto > >>>>>>>> detecting the right one. Make sure the ports like right too. > >>>>>>>> > >>>>>>>>> waitForReplicasToComeUp > >>>>>>>>> INFO: Waiting until we see more replicas up: total=2 found=1 > >>>>>>>>> timeoutin=179999 > >>>>>>>> > >>>>>>>> That happens when you stop the cluster and try to start it again - > >>>>> before > >>>>>>>> a leader is chosen, it will wait for all known replicas fora shard > >>>>> to come > >>>>>>>> up so that everyone can sync up and have a chance to be the best > >>>>> leader. So > >>>>>>>> at this point it was only finding one of 2 known replicas and > >>>>> waiting for > >>>>>>>> the second to come up. After a couple minutes (configurable) it > >>>>> will just > >>>>>>>> continue anyway without the missing replica (if it doesn't show > >> up). > >>>>>>>> > >>>>>>>> - Mark > >>>>>>>> > >>>>>>>> On Dec 5, 2012, at 4:21 PM, Sudhakar Maddineni < > >>>>> maddineni...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi, > >>>>>>>>> We are uploading solr documents to the index in batches using 30 > >>>>> threads > >>>>>>>>> and using ThreadPoolExecutor, LinkedBlockingQueue with max limit > >>>>> set to > >>>>>>>>> 10000. > >>>>>>>>> In the code, we are using HttpSolrServer and add(inputDoc) method > >>>>> to add > >>>>>>>>> docx. > >>>>>>>>> And, we have the following commit settings in solrconfig: > >>>>>>>>> > >>>>>>>>> <autoCommit> > >>>>>>>>> <maxTime>300000</maxTime> > >>>>>>>>> <maxDocs>10000</maxDocs> > >>>>>>>>> <openSearcher>false</openSearcher> > >>>>>>>>> </autoCommit> > >>>>>>>>> > >>>>>>>>> <autoSoftCommit> > >>>>>>>>> <maxTime>1000</maxTime> > >>>>>>>>> </autoSoftCommit> > >>>>>>>>> > >>>>>>>>> Cluster Details: > >>>>>>>>> ---------------------------- > >>>>>>>>> solr version - 4.0 > >>>>>>>>> zookeeper version - 3.4.3 [zookeeper ensemble with 3 nodes] > >>>>>>>>> numshards=2 , > >>>>>>>>> 001, 002, 003 are the solr nodes and these three are behind the > >>>>>>>>> loadbalancer <vip> > >>>>>>>>> 001, 003 assigned to shard1; 002 assigned to shard2 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Logs:Getting the errors in the below sequence after uploading > >> some > >>>>> docx: > >>>>>>>>> > >>>>>>>> > >>>>> > >> > ----------------------------------------------------------------------------------------------------------- > >>>>>>>>> 003 > >>>>>>>>> Dec 4, 2012 12:11:46 PM > >>>>> org.apache.solr.cloud.ShardLeaderElectionContext > >>>>>>>>> waitForReplicasToComeUp > >>>>>>>>> INFO: Waiting until we see more replicas up: total=2 found=1 > >>>>>>>>> timeoutin=179999 > >>>>>>>>> > >>>>>>>>> 001 > >>>>>>>>> Dec 4, 2012 12:12:59 PM > >>>>>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor > >>>>>>>>> doDefensiveChecks > >>>>>>>>> SEVERE: ClusterState says we are the leader, but locally we don't > >>>>> think > >>>>>>>> so > >>>>>>>>> > >>>>>>>>> 003 > >>>>>>>>> Dec 4, 2012 12:12:59 PM org.apache.solr.common.SolrException log > >>>>>>>>> SEVERE: forwarding update to <001>:8080/solr/core1/ failed - > >>>>> retrying ... > >>>>>>>>> > >>>>>>>>> 001 > >>>>>>>>> Dec 4, 2012 12:12:59 PM org.apache.solr.common.SolrException log > >>>>>>>>> SEVERE: Error uploading: org.apache.solr.common.SolrException: > >>>>> Server at > >>>>>>>>> <vip>/solr/core1. returned non ok status:503, message:Service > >>>>> Unavailable > >>>>>>>>> at > >>>>>>>>> > >>>>>>>> > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372) > >>>>>>>>> at > >>>>>>>>> > >>>>>>>> > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) > >>>>>>>>> 001 > >>>>>>>>> Dec 4, 2012 12:25:45 PM org.apache.solr.common.SolrException log > >>>>>>>>> SEVERE: Error while trying to recover. > >>>>>>>>> core=core1:org.apache.solr.common.SolrException: We are not the > >>>>> leader > >>>>>>>>> at > >>>>>>>>> > >>>>>>>> > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:401) > >>>>>>>>> > >>>>>>>>> 001 > >>>>>>>>> Dec 4, 2012 12:44:38 PM org.apache.solr.common.SolrException log > >>>>>>>>> SEVERE: Error uploading: > >>>>>>>> org.apache.solr.client.solrj.SolrServerException: > >>>>>>>>> IOException occured when talking to server at <vip>/solr/core1 > >>>>>>>>> at > >>>>>>>>> > >>>>>>>> > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:413) > >>>>>>>>> at > >>>>>>>>> > >>>>>>>> > >>>>> > >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) > >>>>>>>>> at > >>>>>>>>> > >>>>>>>> > >>>>> > >> > org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) > >>>>>>>>> at > >> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) > >>>>>>>>> ... 5 lines omitted ... > >>>>>>>>> at java.lang.Thread.run(Unknown Source) > >>>>>>>>> Caused by: java.net.SocketException: Connection reset > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> After sometime, all the three servers are going down. > >>>>>>>>> > >>>>>>>>> Appreciate, if someone could let us know what we are missing. > >>>>>>>>> > >>>>>>>>> Thx,Sudhakar. > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> <logs_error.txt> > >>>>> > >>>>> > >>>> > >>> > >> > >
