Re: Doc add limit, im experiencing it too
I sent out an email about this a while back, but basically this limit appears only on Tomcat and only when Solr attempts to write to the response. You can work around it by splitting up your posts so that you're posting less than 5000 (or whatever your limit seems to be) at a time. You DO NOT have to commit after each post. I recently indexed a 38 million document data base with this problem and although it took about 8-9 hours it did work... I only commited every 100,000 or so. -Sangraal On 9/6/06, Michael Imbeault <[EMAIL PROTECTED]> wrote: Old issue (see http://www.mail-archive.com/solr-user@lucene.apache.org/msg00651.html), but I'm experiencing the same exact thing on windows xp, latest tomcat. I noticed that the tomcat process gobbles memory (10 megs a second maybe) and then jams at 125 megs. Can't find a fix yet. I'm using a php interface and curl to post my xml, one document at a time, and commit every 100 document. Indexing 3 docs, it hangs at maybe 5000. Anyone got an idea on this one? It would be helpful. I may try to switch to jetty tomorrow if nothing works :( -- Michael Imbeault CHUL Research Center (CHUQ) 2705 boul. Laurier Ste-Foy, QC, Canada, G1V 4G2 Tel: (418) 654-2705, Fax: (418) 654-2212
Re: Doc add limit, im experiencing it too
: Old issue (see : http://www.mail-archive.com/solr-user@lucene.apache.org/msg00651.html), : but I'm experiencing the same exact thing on windows xp, latest tomcat. did you notice the followup thread from sangraal where he mentioned that he'd narrowed the problem down to both using Tomcat and submitting docs containing CDATA? ... http://www.nabble.com/Add-doc-limit---Follow-Up-tf2186440.html#a6048436 : I noticed that the tomcat process gobbles memory (10 megs a second : maybe) and then jams at 125 megs. Can't find a fix yet. I'm using a php : interface and curl to post my xml, one document at a time, and commit : every 100 document. Indexing 3 docs, it hangs at maybe 5000. Anyone interesting ... you may not be hitting the same problem, Sangraal specificly said he only encountered this bug when submitting a large number of docs in a single request -- when you say it jams at 125 what do you mean? ... are you sure you aren't just getting an OutOfMemory error? -Hoss
Doc add limit, im experiencing it too
Old issue (see http://www.mail-archive.com/solr-user@lucene.apache.org/msg00651.html), but I'm experiencing the same exact thing on windows xp, latest tomcat. I noticed that the tomcat process gobbles memory (10 megs a second maybe) and then jams at 125 megs. Can't find a fix yet. I'm using a php interface and curl to post my xml, one document at a time, and commit every 100 document. Indexing 3 docs, it hangs at maybe 5000. Anyone got an idea on this one? It would be helpful. I may try to switch to jetty tomorrow if nothing works :( -- Michael Imbeault CHUL Research Center (CHUQ) 2705 boul. Laurier Ste-Foy, QC, Canada, G1V 4G2 Tel: (418) 654-2705, Fax: (418) 654-2212
