commit stops
Hi We have an index of about 3.5gb which seems to work fine until it suddenly stops accepting new commits. Users can still search on the front end but nothing new can be committed and it always times out on commit. Any ideas? Thanks in advance 2012-04-27 14:14:20.537:WARN::Committed before 500 null||org.mortbay.jetty.EofException|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)|?at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)|?at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)|?at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)|?at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)|?at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)|?at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:115)|?at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:344)|?at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)|?at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)|?at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)|?at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)|?at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)|?at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)|?at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)|?at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)|?at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?at org.mortbay.jetty.Server.handle(Server.java:326)|?at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)|?at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)|?at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)|Caused by: java.net.SocketException: Broken pipe|?at java.net.SocketOutputStream.socketWrite0(Native Method)|?at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)|?at java.net.SocketOutputStream.write(SocketOutputStream.java:136)|?at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)|?at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129)|?at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161)|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714)|?... 25 more| 2012-04-27 14:14:20.537:WARN::/solr/update/ java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:380) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:283) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 27-Apr-2012 14:14:20 org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278) at
Re: commit stops
One more thing I noticed is the the schema browser in the admin interface also eventually times out… Any ideas from anyone ? From: Mav Peri To: solr-user@lucene.apache.orgmailto:solr-user@lucene.apache.org solr-user@lucene.apache.orgmailto:solr-user@lucene.apache.org Subject: commit stops Hi We have an index of about 3.5gb which seems to work fine until it suddenly stops accepting new commits. Users can still search on the front end but nothing new can be committed and it always times out on commit. Any ideas? Thanks in advance 2012-04-27 14:14:20.537:WARN::Committed before 500 null||org.mortbay.jetty.EofException|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)|?at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)|?at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)|?at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)|?at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)|?at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)|?at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:115)|?at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:344)|?at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)|?at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)|?at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)|?at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)|?at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)|?at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)|?at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)|?at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)|?at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?at org.mortbay.jetty.Server.handle(Server.java:326)|?at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)|?at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)|?at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)|Caused by: java.net.SocketException: Broken pipe|?at java.net.SocketOutputStream.socketWrite0(Native Method)|?at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)|?at java.net.SocketOutputStream.write(SocketOutputStream.java:136)|?at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)|?at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129)|?at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161)|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714)|?... 25 more| 2012-04-27 14:14:20.537:WARN::/solr/update/ java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:380) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:283) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 27-Apr-2012 14:14:20 org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException
Re: commit stops
On Fri, Apr 27, 2012 at 9:18 AM, mav.p...@holidaylettings.co.uk mav.p...@holidaylettings.co.uk wrote: We have an index of about 3.5gb which seems to work fine until it suddenly stops accepting new commits. Users can still search on the front end but nothing new can be committed and it always times out on commit. Any ideas? Perhaps the commit happens to cause a major merge which may take a long time (and solr isn't going to allow overlapping commits). How long does a commit request take to time out? What Solr version is this? Do you have any kind of auto-commit set up? How often are you manually committing? -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
Re: commit stops
Thanks for the reply The client expects a response within 2 minutes and after that will report an error. When we build fresh it seems to work and the operation takes a second or two to complete. Once it gets to a stage it hangs it simply won't accept any further commits. I did an index check and all was ok. I don¹t see any major commit happening at any time, it seems to just hang. Even starting up and shutting down takes ages. We make 3 - 4 commits a day. We use solr 3.5 No autocommit On 28/04/2012 00:56, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Apr 27, 2012 at 9:18 AM, mav.p...@holidaylettings.co.uk mav.p...@holidaylettings.co.uk wrote: We have an index of about 3.5gb which seems to work fine until it suddenly stops accepting new commits. Users can still search on the front end but nothing new can be committed and it always times out on commit. Any ideas? Perhaps the commit happens to cause a major merge which may take a long time (and solr isn't going to allow overlapping commits). How long does a commit request take to time out? What Solr version is this? Do you have any kind of auto-commit set up? How often are you manually committing? -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
Re: commit stops
We also see extreme slowness using Solr 3.6 when trying to commit a delete. We also get hangs. We do 1 commit at most a week. Rebuilding from scratching using DIH works fine and has never hung. Bill Bell Sent from mobile On Apr 27, 2012, at 5:59 PM, mav.p...@holidaylettings.co.uk mav.p...@holidaylettings.co.uk wrote: Thanks for the reply The client expects a response within 2 minutes and after that will report an error. When we build fresh it seems to work and the operation takes a second or two to complete. Once it gets to a stage it hangs it simply won't accept any further commits. I did an index check and all was ok. I don¹t see any major commit happening at any time, it seems to just hang. Even starting up and shutting down takes ages. We make 3 - 4 commits a day. We use solr 3.5 No autocommit On 28/04/2012 00:56, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Apr 27, 2012 at 9:18 AM, mav.p...@holidaylettings.co.uk mav.p...@holidaylettings.co.uk wrote: We have an index of about 3.5gb which seems to work fine until it suddenly stops accepting new commits. Users can still search on the front end but nothing new can be committed and it always times out on commit. Any ideas? Perhaps the commit happens to cause a major merge which may take a long time (and solr isn't going to allow overlapping commits). How long does a commit request take to time out? What Solr version is this? Do you have any kind of auto-commit set up? How often are you manually committing? -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
