Just out of curiosity, are you using sharing the zookeepers between the different versions of Solr? If So, are you specifying a zookeeper chroot? On May 5, 2016 2:05 PM, "Susheel Kumar" <susheel2...@gmail.com> wrote:
> Nick, Hoss - Things are back to normal with ZK 3.4.8 and ZK-6.0.0. I > switched to Solr 5.5.0 with ZK 3.4.8 which worked fine and then installed > 6.0.0. I suspect (not 100% sure) i left ZK dataDir / Solr collection > directory data from previous ZK/solr version which probably was making Solr > 6 in unstable state. > > Thanks, > Susheel > > On Wed, May 4, 2016 at 9:56 PM, Susheel Kumar <susheel2...@gmail.com> > wrote: > > > Thanks, Nick & Hoss. I am using the exact same machine, have wiped out > > solr 5.5.0 and installed solr-6.0.0 with external ZK 3.4.8. I checked > the > > File Description limit for user solr, which is 12000 and increased to > > 52000. Don't see "too many files open..." error now in Solr log but still > > Solr connection getting lost in Admin panel. > > > > Let me do some more tests and install older version back to confirm and > > will share the findings. > > > > Thanks, > > Susheel > > > > On Wed, May 4, 2016 at 8:11 PM, Chris Hostetter < > hossman_luc...@fucit.org> > > wrote: > > > >> > >> : Thanks, Nick. Do we know any suggested # for file descriptor limit > with > >> : Solr6? Also wondering why i haven't seen this problem before with > Solr > >> 5.x? > >> > >> are you running Solr6 on the exact same host OS that you were running > >> Solr5 on? > >> > >> even if you are using the "same OS version" on a diff machine, that > could > >> explain the discrepency if you (or someone else) increased the file > >> descriptor limit on the "old machine" but that neverh appened on the > 'new > >> machine" > >> > >> > >> > >> : On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev < > nick.vasily...@gmail.com > >> > > >> : wrote: > >> : > >> : > It looks like you have too many open files, try increasing the file > >> : > descriptor limit. > >> : > > >> : > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar < > susheel2...@gmail.com> > >> : > wrote: > >> : > > >> : > > Hello, > >> : > > > >> : > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and > >> used > >> : > the > >> : > > install service to setup solr. > >> : > > > >> : > > After launching Solr Admin Panel on server1, it looses connections > >> in few > >> : > > seconds and then comes back and other node server2 is marked as > >> Down in > >> : > > cloud graph. After few seconds its loosing the connection and > comes > >> back. > >> : > > > >> : > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK > >> 3.4.8. > >> : > > Have never seen this error before with solr 5.x with ZK 3.4.6. > >> : > > > >> : > > Below log from server1 & server2. The ZK has 3 nodes with chroot > >> : > enabled. > >> : > > > >> : > > Thanks, > >> : > > Susheel > >> : > > > >> : > > server1/solr.log > >> : > > > >> : > > ==== > >> : > > > >> : > > > >> : > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ] > >> : > > o.a.s.c.c.ZkStateReader path=[/collections/collection1] > >> : > > [configName]=[collection1] specified config exists in ZooKeeper > >> : > > > >> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 > >> QTime=25 > >> : > > > >> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ] > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with > >> params > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true > >> : > > > >> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2 > >> : > > > >> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/cores > >> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0 > >> : > > > >> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/info/system > >> : > > params={wt=json&_=1462389588126} status=0 QTime=25 > >> : > > > >> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ] > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with > >> params > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true > >> : > > > >> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3 > >> : > > > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 > >> QTime=25 > >> : > > > >> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ] > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with > >> params > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true > >> : > > > >> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2 > >> : > > > >> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/cores > >> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0 > >> : > > > >> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/info/system > >> : > > params={wt=json&_=1462389588126} status=0 QTime=25 > >> : > > > >> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ] > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with > >> params > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true > >> : > > > >> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3 > >> : > > > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception > (java.net.SocketException) > >> : > caught > >> : > > when connecting to {}->http://server2:8983: Too many open files > >> : > > > >> : > > … > >> : > > > >> : > > … > >> : > > > >> : > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ] > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}-> > >> http://server2:8983 > >> : > > > >> : > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ] > >> : > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: > >> Error > >> : > > trying to proxy request for url: > >> : > > http://server2:8983/solr/collection1/admin/luke > >> : > > > >> : > > at > >> : > > > >> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588) > >> : > > > >> : > > at > >> : > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) > >> : > > > >> : > > at > >> : > > > >> : > > >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > >> : > > > >> : > > at > >> : > > > >> : > > >> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) > >> : > > > >> : > > > >> : > > > >> : > > > >> : > > > >> : > > server2/solr.log > >> : > > > >> : > > ====== > >> : > > > >> : > > > >> : > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector > >> Started > >> : > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} > >> : > > > >> : > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started > >> @3850ms > >> : > > > >> : > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/cores > >> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34 > >> : > > > >> : > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/zookeeper > >> : > > params={path=/live_nodes&wt=json&_=1462389620113} status=0 > QTime=42 > >> : > > > >> : > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/info/system > >> : > > params={wt=json&_=1462389620002} status=0 QTime=50 > >> : > > > >> : > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/zookeeper > >> : > > > >> : > > > >> : > > >> > params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113} > >> : > > status=0 QTime=6 > >> : > > > >> : > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ] > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with > >> params > >> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true > >> : > > > >> : > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ] > >> : > o.a.s.s.HttpSolrCall > >> : > > [admin] webapp=null path=/admin/collections > >> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19 > >> : > > > >> : > > 2016-05-04 19:34:35.760 WARN > >> : > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector > >> : > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ] > >> : > > o.e.j.s.ServerConnector > >> : > > > >> : > > java.io.IOException: Too many open files > >> : > > > >> : > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native > Method) > >> : > > > >> : > > at > >> : > > > >> : > > >> > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > >> : > > > >> : > > at > >> : > > > >> : > > >> > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > >> : > > > >> : > > at > >> : > > > >> > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) > >> : > > > >> : > > at > >> : > > > >> : > > > >> : > > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > >> : > > > >> : > > at java.lang.Thread.run(Thread.java:745) > >> : > > > >> : > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ] > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores > >> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0 > >> : > > > >> : > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ] > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging > >> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4 > >> : > > > >> : > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ] > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system > >> : > > params={wt=json&_=1462389620002} status=0 QTime=99 > >> : > > > >> : > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ] > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with > >> params > >> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true > >> : > > > >> : > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ] > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections > >> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2 > >> : > > > >> : > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ] > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging > >> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0 > >> : > > > >> : > > >> : > >> > >> -Hoss > >> http://www.lucidworks.com/ > > > > > > >