Re: Debugging update request
@Michael: Yep, that's the bit that's addressed by the two patches I referenced. If you can try this with 4.5 (or the soon to be done 4.5.1), the problem should go away. @Chris: I think you have a different issue. A very quick glance at your stack trace doesn't really show anything outstanding. There are always a bunch of threads waiting around for something to do that show up as blocked. So I'm pretty puzzled. Are your Solr logs showing anything when you try to update after this occurs? On Wed, Oct 16, 2013 at 11:32 AM, Chris Geeringh geeri...@gmail.com wrote: Here is my jstack output... Lots of blocked threads. http://pastebin.com/1ktjBYbf On 16 October 2013 10:28, michael.boom my_sky...@yahoo.com wrote: I got the trace from jstack. I found references to semaphore but not sure if this is what you meant. Here's the trace: http://pastebin.com/15QKAz7U -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
Thanks Erick! The version is 4.4.0. I'm posting 100k docs batches every 30-40 sec from each indexing client and sometimes two or more clients post in a very small timeframe. That's when i think the deadlock happens. I'll try to replicate the problem and check the thread dump. -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095821.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
I ran an import last night, and this morning my cloud wouldn't accept updates. I'm running the latest 4.6 snapshot. I was importing with latest solrj snapshot, and using java bin transport with CloudSolrServer. The cluster had indexed ~1.3 million docs before no further updates were accepted, querying still working. I'll run jstack shortly and provide the results. On Wednesday, October 16, 2013, michael.boom wrote: Thanks Erick! The version is 4.4.0. I'm posting 100k docs batches every 30-40 sec from each indexing client and sometimes two or more clients post in a very small timeframe. That's when i think the deadlock happens. I'll try to replicate the problem and check the thread dump. -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095821.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
I got the trace from jstack. I found references to semaphore but not sure if this is what you meant. Here's the trace: http://pastebin.com/15QKAz7U -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
Here is my jstack output... Lots of blocked threads. http://pastebin.com/1ktjBYbf On 16 October 2013 10:28, michael.boom my_sky...@yahoo.com wrote: I got the trace from jstack. I found references to semaphore but not sure if this is what you meant. Here's the trace: http://pastebin.com/15QKAz7U -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html Sent from the Solr - User mailing list archive at Nabble.com.
Debugging update request
As a followup to another thread, where I described how my SolrCloud sometimes just stops accepting updates http://lucene.472066.n3.nabble.com/Concurent-indexing-td4095409.html I have a question, is there a way to debug or analyze the update request? Verbose output or anything else? It happens for me that when in the above situation, I'm using the post.sh tool to post 1 doc and i get no feedback, it just hangs and waits. 10x! -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
What version of Solr? There's a known issue with distributed deadlock (since fixed) where under heavy load from multiple clients to multiple Solr nodes, you can get into a situation where it will stop accepting updates, but _will_ continue to serve queries. If this describes your situation, dump the stack (jstack works) and look for the word semaphore in the stack traces. If Solr is waiting on that, that's your problem. SOLR-4816 (4.5) should alleviate this if you're indexing from SolrJ, and SOLR-5216 should fix it on the server. Best, Erick On Tue, Oct 15, 2013 at 10:54 AM, maephisto my_sky...@yahoo.com wrote: As a followup to another thread, where I described how my SolrCloud sometimes just stops accepting updates http://lucene.472066.n3.nabble.com/Concurent-indexing-td4095409.html I have a question, is there a way to debug or analyze the update request? Verbose output or anything else? It happens for me that when in the above situation, I'm using the post.sh tool to post 1 doc and i get no feedback, it just hangs and waits. 10x! -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619.html Sent from the Solr - User mailing list archive at Nabble.com.
