Re: overseer queue clogged

2014-05-29 Thread ryan.cooke
We were running Solr 4.2, and are in the process of upgrading. I believe that the particular scenario that was clogging our queue was resolved in 4.7.1 - https://issues.apache.org/jira/browse/SOLR-5811 -- View this message in context: http://lucene.472066.n3.nabble.com/overseer-queue-clogged

Re: overseer queue clogged

2014-05-22 Thread James Hardwick
at 11:19:54 AM, ryan.cooke (ryan.co...@gmail.com > (mailto:ryan.co...@gmail.com)) wrote: > > I saw an overseer queue clogged as well due to a bad message in the queue. > Unfortunately this went unnoticed for a while until there were 130K messages > in the overseer queue. Since it w

Re: overseer queue clogged

2014-05-01 Thread Mark Miller
the leader, but locally we don't think so" Bringing up new replicas time out when attempting to fetch shard id -- View this message in context: http://lucene.472066.n3.nabble.com/overseer-queue-clogged-tp4047878p4134129.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: overseer queue clogged

2014-05-01 Thread ryan.cooke
I saw an overseer queue clogged as well due to a bad message in the queue. Unfortunately this went unnoticed for a while until there were 130K messages in the overseer queue. Since it was a production system we were not able to simply stop everything and delete all Zookeeper data, so we manually

Re: overseer queue clogged

2013-03-22 Thread Mark Miller
On Mar 22, 2013, at 5:54 PM, Gary Yngve wrote: > Thanks, Mark! > > The core node names in the solr.xml in solr4.2 is great! Maybe in 4.3 it > can be supported via API? It is with the core admin api - do you mean the collections api? Please make a JIRA for any feature requests so they don't g

Re: overseer queue clogged

2013-03-22 Thread Gary Yngve
Thanks, Mark! The core node names in the solr.xml in solr4.2 is great! Maybe in 4.3 it can be supported via API? Also I am glad you mentioned in other post the chance to namespace zookeeper by adding a path to the end of the comma-delim zk hosts. That works out really well in our situation for

Re: overseer queue clogged

2013-03-16 Thread Mark Miller
Yeah, I don't know that I've ever tried with 4.0, but I've done this with 4.1 and 4.2. - Mark On Mar 16, 2013, at 12:19 PM, Gary Yngve wrote: > Cool, I'll need to try this. I could have sworn that it didn't work that > way in 4.0, but maybe my test was bunk. > > -g > > > On Fri, Mar 15, 20

Re: overseer queue clogged

2013-03-16 Thread Gary Yngve
Cool, I'll need to try this. I could have sworn that it didn't work that way in 4.0, but maybe my test was bunk. -g On Fri, Mar 15, 2013 at 9:41 PM, Mark Miller wrote: > > You can do this - just modify your starting Solr example to have no cores > in solr.xml. You won't be able to make use of

Re: overseer queue clogged

2013-03-15 Thread Mark Miller
On Mar 16, 2013, at 12:30 AM, Gary Yngve wrote: > I will upgrade to 4.2 this weekend and see what happens. We are on ec2 and > have had a few issues with hostnames with both zk and solr. (but in this > case i haven't rebooted any instances either) There is actually a new feature in 4.2 that le

Re: overseer queue clogged

2013-03-15 Thread Gary Yngve
I will upgrade to 4.2 this weekend and see what happens. We are on ec2 and have had a few issues with hostnames with both zk and solr. (but in this case i haven't rebooted any instances either) it's relatively a pain to do the upgrade because we have a query/scorer fork of lucene along with suppl

Re: overseer queue clogged

2013-03-15 Thread Mark Miller
On Mar 15, 2013, at 10:04 PM, Gary Yngve wrote: > i think those followers are red from trying to forward requests to the > overseer while it was being restarted. i guess i'll see if they become > green over time. or i guess i can restart them one at a time.. Restarting the cluster clear thing

Re: overseer queue clogged

2013-03-15 Thread Mark Miller
It looks like they are not picking up the new leader state for some reason… Thats where where it say the local state doesn't match the zookeeper state. If the local state doesn't match the zookeeper state in a short amount of time when a new leader comes, everything will bail because it assumes

Re: overseer queue clogged

2013-03-15 Thread Gary Yngve
i think those followers are red from trying to forward requests to the overseer while it was being restarted. i guess i'll see if they become green over time. or i guess i can restart them one at a time.. On Fri, Mar 15, 2013 at 6:53 PM, Gary Yngve wrote: > it doesn't appear to be a shard1 vs

Re: overseer queue clogged

2013-03-15 Thread Gary Yngve
it doesn't appear to be a shard1 vs shard11 issue... 60% of my followers are red now in the solr cloud graph.. trying to figure out what that means... On Fri, Mar 15, 2013 at 6:48 PM, Gary Yngve wrote: > I restarted the overseer node and another took over, queues are empty now. > > the server w

Re: overseer queue clogged

2013-03-15 Thread Gary Yngve
I restarted the overseer node and another took over, queues are empty now. the server with core production_things_shard1_2 is having these errors: shard update error RetryNode: http://10.104.59.189:8883/solr/production_things_shard11_replica1/:org.apache.solr.client.solrj.SolrServerException: Ser

Re: overseer queue clogged

2013-03-15 Thread Mark Miller
Strange - we hardened that loop in 4.1 - so I'm not sure what happened here. Can you do a stack dump on the overseer and see if you see an Overseer thread running perhaps? Or just post the results? To recover, you should be able to just restart the Overseer node and have someone else take over

Re: overseer queue clogged

2013-03-15 Thread Gary Yngve
Also, looking at overseer_elect, everything looks fine. node is valid and live. On Fri, Mar 15, 2013 at 4:47 PM, Gary Yngve wrote: > Sorry, should have specified. 4.1 > > > > > On Fri, Mar 15, 2013 at 4:33 PM, Mark Miller wrote: > >> What Solr version? 4.0, 4.1 4.2? >> >> - Mark >> >> On Mar

Re: overseer queue clogged

2013-03-15 Thread Gary Yngve
Sorry, should have specified. 4.1 On Fri, Mar 15, 2013 at 4:33 PM, Mark Miller wrote: > What Solr version? 4.0, 4.1 4.2? > > - Mark > > On Mar 15, 2013, at 7:19 PM, Gary Yngve wrote: > > > my solr cloud has been running fine for weeks, but about a week ago, it > > stopped dequeueing from th

Re: overseer queue clogged

2013-03-15 Thread Mark Miller
What Solr version? 4.0, 4.1 4.2? - Mark On Mar 15, 2013, at 7:19 PM, Gary Yngve wrote: > my solr cloud has been running fine for weeks, but about a week ago, it > stopped dequeueing from the overseer queue, and now there are thousands of > tasks on the queue, most which look like > > { > "ope