: Thanks, Nick. Do we know any suggested # for file descriptor limit with
: Solr6? Also wondering why i haven't seen this problem before with Solr 5.x?
are you running Solr6 on the exact same host OS that you were running
Solr5 on?
even if you are using the "same OS version" on a diff machine, that could
explain the discrepency if you (or someone else) increased the file
descriptor limit on the "old machine" but that neverh appened on the 'new
machine"
: On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <nick.vasily...@gmail.com>
: wrote:
:
: > It looks like you have too many open files, try increasing the file
: > descriptor limit.
: >
: > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <susheel2...@gmail.com>
: > wrote:
: >
: > > Hello,
: > >
: > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used
: > the
: > > install service to setup solr.
: > >
: > > After launching Solr Admin Panel on server1, it looses connections in few
: > > seconds and then comes back and other node server2 is marked as Down in
: > > cloud graph. After few seconds its loosing the connection and comes back.
: > >
: > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
: > > Have never seen this error before with solr 5.x with ZK 3.4.6.
: > >
: > > Below log from server1 & server2. The ZK has 3 nodes with chroot
: > enabled.
: > >
: > > Thanks,
: > > Susheel
: > >
: > > server1/solr.log
: > >
: > > ====
: > >
: > >
: > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
: > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
: > > [configName]=[collection1] specified config exists in ZooKeeper
: > >
: > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
: > >
: > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
: > >
: > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389588126} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
: > >
: > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
: > >
: > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389588126} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > …
: > >
: > > …
: > >
: > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
: > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
: > > trying to proxy request for url:
: > > http://server2:8983/solr/collection1/admin/luke
: > >
: > > at
: > > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
: > >
: > > at
: > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
: > >
: > > at
: > >
: > >
: >
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
: > >
: > > at
: > >
: > >
: >
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
: > >
: > > at
: > >
: > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
: > >
: > > at
: > >
: > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
: > >
: > >
: > >
: > >
: > >
: > > server2/solr.log
: > >
: > > ======
: > >
: > >
: > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector Started
: > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
: > >
: > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started @3850ms
: > >
: > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
: > >
: > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/zookeeper
: > > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
: > >
: > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389620002} status=0 QTime=50
: > >
: > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/zookeeper
: > >
: > >
: >
params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
: > > status=0 QTime=6
: > >
: > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
: > >
: > > 2016-05-04 19:34:35.760 WARN
: > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
: > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
: > > o.e.j.s.ServerConnector
: > >
: > > java.io.IOException: Too many open files
: > >
: > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
: > >
: > > at
: > >
: > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
: > >
: > > at
: > >
: > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
: > >
: > > at
: > > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
: > >
: > > at
: > >
: > >
: >
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
: > >
: > > at java.lang.Thread.run(Thread.java:745)
: > >
: > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
: > >
: > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
: > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
: > >
: > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389620002} status=0 QTime=99
: > >
: > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
: > >
: > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
: > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
: > >
: >
:
-Hoss
http://www.lucidworks.com/
