The updates are pushed in real time not batched. No complex analysis and everything is committed using autocommit settings in solr.
Thanks Jay Potharaju On Mon, May 7, 2018 at 4:00 PM, Emir Arnautović < emir.arnauto...@sematext.com> wrote: > How do you send documents? Large batches? Complex analysis? Do you send all > batches to the same node? How do you commit? Do you delete by query while > indexing? > > Emir > > On Tue, May 8, 2018, 12:30 AM Jay Potharaju <jspothar...@gmail.com> wrote: > > > I didn't see any OOM errors in the logs on either of the nodes. I saw GC > > pause of 1 second on the box that was throwing error ...but nothing on > the > > other node. Any other recommendations? > > Thanks > > > > > > Thanks > > Jay Potharaju > > > > > > On Mon, May 7, 2018 at 9:48 AM, Jay Potharaju <jspothar...@gmail.com> > > wrote: > > > > > Ah thanks for explaining that! > > > > > > Thanks > > > Jay Potharaju > > > > > > > > > On Mon, May 7, 2018 at 9:45 AM, Emir Arnautović < > > > emir.arnauto...@sematext.com> wrote: > > > > > >> Node A receives batch of documents to index. It forwards documents to > > >> shards that are on the node B. Node B is having issues with GC so it > > takes > > >> a while to respond. Node A sees it as read timeout and reports it in > > logs. > > >> So the issue is on node B not node A. > > >> > > >> Emir > > >> -- > > >> Monitoring - Log Management - Alerting - Anomaly Detection > > >> Solr & Elasticsearch Consulting Support Training - > http://sematext.com/ > > >> > > >> > > >> > > >> > On 7 May 2018, at 18:39, Jay Potharaju <jspothar...@gmail.com> > wrote: > > >> > > > >> > Yes, the nodes are well balanced. I am just using these boxes for > > >> indexing > > >> > the data and is not serving any traffic at this time. The error > > >> indicates > > >> > it is having issues errors on the shards that are hosted on the box > > and > > >> not > > >> > on the other box. > > >> > I will check GC logs to see if there were any issues. > > >> > thanks > > >> > > > >> > Thanks > > >> > Jay Potharaju > > >> > > > >> > > > >> > On Mon, May 7, 2018 at 9:34 AM, Emir Arnautović < > > >> > emir.arnauto...@sematext.com> wrote: > > >> > > > >> >> Hi Jay, > > >> >> My first guess would be that there was some major GC on other box > so > > it > > >> >> did not respond on time. Are your nodes well balanced - do they > serve > > >> equal > > >> >> amount of data? > > >> >> > > >> >> Thanks, > > >> >> Emir > > >> >> -- > > >> >> Monitoring - Log Management - Alerting - Anomaly Detection > > >> >> Solr & Elasticsearch Consulting Support Training - > > >> http://sematext.com/ > > >> >> > > >> >> > > >> >> > > >> >>> On 7 May 2018, at 18:11, Jay Potharaju <jspothar...@gmail.com> > > wrote: > > >> >>> > > >> >>> Hi, > > >> >>> I am seeing the following lines in the error log. My setup has 2 > > >> nodes in > > >> >>> the solrcloud cluster, each node has 3 shards with no replication. > > >> From > > >> >> the > > >> >>> error log it seems like all the shards on this box are throwing > > async > > >> >>> exception errors. Other node in the cluster does not have any > errors > > >> in > > >> >> the > > >> >>> logs. Any suggestions on how to tackle this error? > > >> >>> > > >> >>> Solr setup > > >> >>> Solr:6.6.3 > > >> >>> 2Nodes: 3 shards each > > >> >>> > > >> >>> > > >> >>> ERROR org.apache.solr.servlet.HttpSolrCall > [test_shard3_replica1] ? > > >> >>> null:org.apache.solr.update.processor.DistributedUpdateProcessor$ > > >> >> DistributedUpdatesAsyncException: > > >> >>> Async exception during distributed update: Read timed out > > >> >>> at > > >> >>> > > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish( > > >> >> DistributedUpdateProcessor.java:972) > > >> >>> at > > >> >>> org.apache.solr.update.processor.DistributedUpdateProcessor. > finish( > > >> >> DistributedUpdateProcessor.java:1911) > > >> >>> at > > >> >>> org.apache.solr.handler.ContentStreamHandlerBase. > handleRequestBody( > > >> >> ContentStreamHandlerBase.java:78) > > >> >>> at > > >> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest( > > >> >> RequestHandlerBase.java:173) > > >> >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) > > >> >>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall. > > >> java:723) > > >> >>> at org.apache.solr.servlet.HttpSolrCall.call( > HttpSolrCall.java:529) > > >> >>> at > > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > > >> >> SolrDispatchFilter.java:361) > > >> >>> at > > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > > >> >> SolrDispatchFilter.java:305) > > >> >>> at > > >> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. > > >> >> doFilter(ServletHandler.java:1691) > > >> >>> at > > >> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle( > > >> >> ServletHandler.java:582) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > > >> >> ScopedHandler.java:143) > > >> >>> at > > >> >>> org.eclipse.jetty.security.SecurityHandler.handle( > > >> >> SecurityHandler.java:548) > > >> >>> at > > >> >>> org.eclipse.jetty.server.session.SessionHandler. > > >> >> doHandle(SessionHandler.java:226) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.ContextHandler. > > >> >> doHandle(ContextHandler.java:1180) > > >> >>> at org.eclipse.jetty.servlet.ServletHandler.doScope( > > >> >> ServletHandler.java:512) > > >> >>> at > > >> >>> org.eclipse.jetty.server.session.SessionHandler. > > >> >> doScope(SessionHandler.java:185) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.ContextHandler. > > >> >> doScope(ContextHandler.java:1112) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > > >> >> ScopedHandler.java:141) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( > > >> >> ContextHandlerCollection.java:213) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.HandlerCollection. > > >> >> handle(HandlerCollection.java:119) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > >> >> HandlerWrapper.java:134) > > >> >>> at > > >> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( > > >> >> RewriteHandler.java:335) > > >> >>> at > > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > >> >> HandlerWrapper.java:134) > > >> >>> at org.eclipse.jetty.server.Server.handle(Server.java:534) > > >> >>> at org.eclipse.jetty.server.HttpChannel.handle( > HttpChannel.java:320) > > >> >>> at > > >> >>> org.eclipse.jetty.server.HttpConnection.onFillable( > > >> >> HttpConnection.java:251) > > >> >>> at > > >> >>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > > >> >> AbstractConnection.java:273) > > >> >>> at org.eclipse.jetty.io.FillInterest.fillable( > FillInterest.java:95) > > >> >>> at > > >> >>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run( > > >> >> SelectChannelEndPoint.java:93) > > >> >>> at > > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > > >> >> QueuedThreadPool.java:671) > > >> >>> at > > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > > >> >> QueuedThreadPool.java:589) > > >> >>> at java.lang.Thread.run(Unknown Source) > > >> >>> > > >> >>> > > >> >>> Thanks > > >> >>> Jay > > >> >> > > >> >> > > >> > > >> > > > > > >
