The DEV list is *not* the place for questions. Please have some patience and you will get an answer.
In the meantime, please review https://wiki.apache.org/solr/UsingMailingLists <https://wiki.apache.org/solr/UsingMailingLists> and update this email thread with more information about your system, such as version, copy of some error logs, screenshot of Cloud tab in Admin etc -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 21. jun. 2017 kl. 04.19 skrev S G <sg.online.em...@gmail.com>: > > Got no response on the solr-user mailing list and so trying the dev-mailing > list. > > Please guide me if this should not be done. But I thought that the issue > looks strange enough to post it here. > > Thanks > SG > > > On Mon, Jun 19, 2017 at 8:13 PM, S G <sg.online.em...@gmail.com > <mailto:sg.online.em...@gmail.com>> wrote: > Hi, > > We are stuck in a strange problem. > Whole cluster is red. All nodes are being shown as down. > Restart of the nodes is not helping either. > > All our nodes seem to have gone into a distributed lock. > Here is the grep command I ran on all the solr.log files: > grep "Error trying to proxy request" $f | cut -d" " -f14 | sort | uniq -c > > And the output from 10 different solr-nodes' solr.log file is shown below: > (Basically each node is calling admin/system on other nodes and throwing > exceptions. You can see number of exceptions thrown by each server for every > other server). > > > > SVR_1.log > 13 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 18 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 19 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 15 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 13 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 13 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_2.log > 335 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 23 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 21 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 23 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 23 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_3.log > 24 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 14 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 13 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 14 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 16 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_4.log > 11 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 29 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 7 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 16 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 11 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_5.log > 18 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 16 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 13 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 12 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 16 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 11 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_6.log > 44 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 296 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 40 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 15 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 15 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_7.log > 59 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 215 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 62 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 47 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 61 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_8.log > 13 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 18 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 10 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 7 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 12 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 13 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > SVR_9.log > 38 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 229 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 15 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 22 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 26 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > > SVR_10.log > 9 http://SVR_2:8983/solr/my-collection/admin/system > <http://svr_2:8983/solr/my-collection/admin/system> > 22 http://SVR_3:8983/solr/my-collection/admin/system > <http://svr_3:8983/solr/my-collection/admin/system> > 18 http://SVR_4:8983/solr/my-collection/admin/system > <http://svr_4:8983/solr/my-collection/admin/system> > 14 http://SVR_6:8983/solr/my-collection/admin/system > <http://svr_6:8983/solr/my-collection/admin/system> > 18 http://SVR_7:8983/solr/my-collection/admin/system > <http://svr_7:8983/solr/my-collection/admin/system> > 10 http://SVR_9:8983/solr/my-collection/admin/system > <http://svr_9:8983/solr/my-collection/admin/system> > > > Thanks > SG >
