Thanks for you help Yonik, I've responded to your questions below:
On 7/26/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
It's possible it's not hanging, but just takes a long time on a
specific add. This is because Lucene will occasionally merge
segments. When very large segments are merged, it can take a long
time.
I've left it running (hung) for up to a half hour at a time and I've
verified that my cpu idles during the hang. I have witnessed much shorter
hangs on the ramp up to my 6,144 limit but they have been more like 2 - 10
seconds in length. Perhaps this is the Lucene merging you mentioned.
In the log file, add commands are followed by the number of
milliseconds the operation took. Next time Solr hangs, wait for a
number of minutes until you see the operation logged and note how long
it took.
Here are the last 5 log entries before the hang the last one is doc #6,144.
Also it looks like Tomcat is trying to redeploy the webapp those last tomcat
entries repeat indefinitely every 10 seconds or so. Perhaps this is a Tomcat
problem?
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110705) 0 36596
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110700) 0 36600
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110688) 0 36603
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110690) 0 36608
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110686) 0 36611
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] redeploy resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] redeploy resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT/META-INF/context.xml
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT/WEB-INF/web.xml
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT/META-INF/context.xml
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
/conf/context.xml
How many documents are in the index before you do a batch that causes
a hang? Does it happen on the first batch? If so, you might be
seeing some other bug. What appserver are you using? Do the admin
pages respond when you see this hang? If so, what does a stack trace
look like?
I actually don't think I had the problem on the first batch, in fact my
first batch contained very close to 6,144 documents so perhaps there is a
relation there. Right now, I'm adding to an index with close to 90,000
documents in it.
I'm running Tomcat 5.5.17 and the admin pages respond just fine when it's
hung... I did a thread dump and this is the trace of my update:
"http-8080-Processor25" Id=33 in RUNNABLE (running in native) total cpu
time=6330.7360ms user time=5769.5920ms
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.PrintStream.write(PrintStream.java:412)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java
:112)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:533)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(
HttpURLConnection.java:410)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(
HttpURLConnection.java:934)
at com.gawker.solr.update.GanjaUpdate.doUpdate(GanjaUpdate.java:169)
at com.gawker.solr.update.GanjaUpdate.update(GanjaUpdate.java:62)
at org.apache.jsp.update_jsp._jspService(update_jsp.java:57)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.jasper.servlet.JspServletWrapper.service(
JspServletWrapper.java:332)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java
:314)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(
Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:613)
-Yonik
On 7/26/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
> Hey there... I'm having an issue with large doc updates on my solr
> installation. I'm adding in batches between 2-20,000 docs at a time and
I've
> noticed Solr seems to hang at 6,144 docs every time. Breaking the adds
into
> smaller batches works just fine, but I was wondering if anyone knew why
this
> would happen. I've tried doubling memory as well as tweaking various
config
> options but nothing seems to let me break the 6,144 barrier.
>
> This is the output from Solr admin. Any help would be greatly
appreciated.
>
>
> *name: * updateHandler *class: *
> org.apache.solr.update.DirectUpdateHandler2 *version: * 1.0
*description:
> * Update handler that efficiently directly updates the on-disk main
lucene
> index *stats: *commits : 0
> optimizes : 0
> docsPending : 6144
> deletesPending : 6144
> adds : 6144
> deletesById : 0
> deletesByQuery : 0
> errors : 0
> cumulative_adds : 6144
> cumulative_deletesById : 0
> cumulative_deletesByQuery : 0
> cumulative_errors : 0
> docsDeleted : 0
>
>
