Ali, 20 sounds a way much. Why woun't you start from the single thread, check that everything is correct, then steadily increase number of threads, checking that cluster utilization also increases with every step until you find a number which saturate Solr well, but won't be greater than necessary?
On Sun, Dec 13, 2015 at 12:24 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Dear Emir, > Hi, > Actually Solr is in a deadlock state it will not accept any new document. > (some of them will store in tlog and some of them not) However, It will > response to the new query requests very slowly. Unfortunately right now I > have not any access to full thread dump. But, as I mentioned, it is full of > thread in blocked state. > > P.S: I am using 20 threads for the indexing part. I am suspicious of auto > hard commit part. Since the indexing/updating part is really slow for the > sake of complicate analyzer, it is possible that updating 2500 documents > takes more than 120 seconds so before finishing the first hard commit > second hard commit would arrive same for third and forth and so on. > Therefore it might possible that lots of commit thread would be active at > the same time with lots of documents in memory that are not flushed to disk > yet. However, I am not sure that such scenario could take Solr threads to > deadlock state! > Best regards. > > On Fri, Dec 11, 2015 at 1:02 PM, Emir Arnautovic < > emir.arnauto...@sematext.com> wrote: > > > Hi Ali, > > Is Solr busy at that time and eventually recover or it is deadlocked? Can > > you provide full thread dump when it happened? > > Do you run only indexing at that time? Is "unavailable" only from > indexing > > perspective, or you cannot do anything with Solr? > > Is there any indexing scenario that does not cause this (extreme/useless > > one is without commits)? > > Did you try throttling indexing or changing bulk size? > > How many indexing threads? > > > > Thanks, > > Emir > > > > > > On 11.12.2015 10:06, Ali Nazemian wrote: > > > >> I really appreciate if somebody can help me to solve this problem. > >> Regards. > >> > >> On Tue, Dec 8, 2015 at 9:22 PM, Ali Nazemian <alinazem...@gmail.com> > >> wrote: > >> > >> I did that already. The situation was worse. The autocommit part makes > >>> solr unavailable. > >>> On Dec 8, 2015 7:13 PM, "Emir Arnautovic" < > emir.arnauto...@sematext.com> > >>> wrote: > >>> > >>> Hi Ali, > >>>> Can you try without explicit commits and see if threads will still be > >>>> blocked. > >>>> > >>>> Thanks, > >>>> Emir > >>>> > >>>> On 08.12.2015 16:19, Ali Nazemian wrote: > >>>> > >>>> The indexing load is as follows: > >>>>> - Around 1000 documents every 5 mins. > >>>>> - The indexing speed is slow because of the complicated analyzer > which > >>>>> is > >>>>> applied to each document. It takes around 60 seconds to index 1000 > >>>>> documents with applying this analyzer (It is really slow. However, > >>>>> based > >>>>> on > >>>>> the analyzing part I think it would be acceptable). > >>>>> - The concurrentsolrclient is used in all the indexing/updating > cases. > >>>>> > >>>>> Regards. > >>>>> > >>>>> On Tue, Dec 8, 2015 at 6:36 PM, Ali Nazemian <alinazem...@gmail.com> > >>>>> wrote: > >>>>> > >>>>> Dear Emir, > >>>>> > >>>>>> Hi, > >>>>>> There are some cases that I have soft commit in my application. > >>>>>> However, > >>>>>> the bulk update part has only hard commit for a bulk of 2500 > >>>>>> documents. > >>>>>> Here are some information about the whole indexing/updating > scenarios: > >>>>>> - Indexing part uses soft commit. > >>>>>> - In a single update cases soft commit is used. > >>>>>> - For bulk update batch hard commit is used (on 2500 documents) > >>>>>> - Auto hard commit :120 sec > >>>>>> - Auto soft commit: disable > >>>>>> > >>>>>> Best regards. > >>>>>> > >>>>>> > >>>>>> On Tue, Dec 8, 2015 at 12:35 PM, Emir Arnautovic < > >>>>>> emir.arnauto...@sematext.com> wrote: > >>>>>> > >>>>>> Hi Ali, > >>>>>> > >>>>>>> This thread is blocked because cannot obtain update lock - in this > >>>>>>> particular case when doing soft commit. I am guessing that there > >>>>>>> others are > >>>>>>> blocked for the same reason. Can you tell us bit more about your > >>>>>>> setup > >>>>>>> and > >>>>>>> indexing load and procedure? Do you do explicit commits? > >>>>>>> > >>>>>>> Regards, > >>>>>>> Emir > >>>>>>> > >>>>>>> -- > >>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log > >>>>>>> Management > >>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 08.12.2015 08:16, Ali Nazemian wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>>> There is a while since I have had problem with Solr 5.2.1 and I > >>>>>>>> could > >>>>>>>> not > >>>>>>>> fix it yet. The only think that is clear to me is when I send bulk > >>>>>>>> update > >>>>>>>> to Solr the commit thread will be blocked! Here is the thread dump > >>>>>>>> output: > >>>>>>>> > >>>>>>>> "qtp595445781-8207" prio=10 tid=0x00007f0bf68f5800 nid=0x5785 > >>>>>>>> waiting > >>>>>>>> for > >>>>>>>> monitor entry [0x00007f081cf04000] > >>>>>>>> java.lang.Thread.State: BLOCKED (on object monitor) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:608) > >>>>>>>> - waiting to lock <0x000000067ba2e660> (a java.lang.Object) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1635) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1612) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:161) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:270) > >>>>>>>> at > org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > >>>>>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > >>>>>>>> at > >>>>>>>> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > >>>>>>>> at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > >>>>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:497) > >>>>>>>> at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > >>>>>>>> at > >>>>>>>> org.eclipse.jetty.io > >>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:540) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > >>>>>>>> at > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > >>>>>>>> at java.lang.Thread.run(Thread.java:745) > >>>>>>>> > >>>>>>>> Locked ownable synchronizers: > >>>>>>>> - None > >>>>>>>> > >>>>>>>> FYI there are lots of blocked thread in thread dump report and > Solr > >>>>>>>> becomes > >>>>>>>> really slow in this case. The temporary solution would be > restarting > >>>>>>>> Solr. > >>>>>>>> But, I am really sick of restarting! I really appreciate if > somebody > >>>>>>>> can > >>>>>>>> help me to solve this problem? > >>>>>>>> > >>>>>>>> Best regards. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>> A.Nazemian > >>>>>> > >>>>>> > >>>>>> > >>>>> -- > >>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management > >>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>> > >>>> > >>>> > >> > > -- > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > -- > A.Nazemian > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
