Re: StreamingUpdateSolrServer hangs

2010-04-30 Thread Yonik Seeley
On Thu, Apr 29, 2010 at 7:51 PM, Yonik Seeley yo...@lucidimagination.com wrote: I'm trying to reproduce now... single thread adding documents to a multithreaded client, StreamingUpdateSolrServer(addr,32,4) I'm currently at the 2.5 hour mark and 100M documents - no issues so far. I let it go

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Yonik Seeley
On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Is your application also using

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Lance Norskog
In solrconfig.xml, there is a parameter controlling remote streaming: requestDispatcher handleSelect=true !--Make sure your system has some authentication before enabling remote streaming! -- requestParsers enableRemoteStreaming=true multipartUploadLimitInKB=2048000 / 1) Is this

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Yonik Seeley
On Thu, Apr 29, 2010 at 6:04 PM, Lance Norskog goks...@gmail.com wrote: In solrconfig.xml, there is a parameter controlling remote streaming:   requestDispatcher handleSelect=true      !--Make sure your system has some authentication before enabling remote streaming!  --      requestParsers

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Lance Norskog
What is the garbage collection status when this happens? What are the open sockets in the OS when this happens? Run 'netstat -an | fgrep 8983' where 8983 is the Solr incoming port number. A side note on sockets: SUSS uses the MultiThreadedHttpConnectionManager but never calls

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Yonik Seeley
I'm trying to reproduce now... single thread adding documents to a multithreaded client, StreamingUpdateSolrServer(addr,32,4) I'm currently at the 2.5 hour mark and 100M documents - no issues so far. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague On Thu, Apr 29, 2010 at 5:12 PM,

Re: StreamingUpdateSolrServer hangs

2010-04-16 Thread Sascha Szott
Hi Yonik, Yonik Seeley wrote: Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: I'm using the latest trunk version (rev. 934846) and constantly running into the

Re: StreamingUpdateSolrServer hangs

2010-04-16 Thread Yonik Seeley
Thanks for the report Sascha. So after the hang, it never recovers? Some amount of hanging could be visible if there was a commit on the Solr server or something else to cause the solr requests to block for a while... but it should return to normal on it's own... Looking at the stack trace, it

Re: StreamingUpdateSolrServer hangs

2010-04-16 Thread Sascha Szott
Hi Yonik, thanks for your fast reply. Yonik Seeley wrote: Thanks for the report Sascha. So after the hang, it never recovers? Some amount of hanging could be visible if there was a commit on the Solr server or something else to cause the solr requests to block for a while... but it should

Re: StreamingUpdateSolrServer hangs

2010-04-16 Thread Rich Cariens
I experienced the hang described with the Solr 1.4.0 build. Yonik - I also thought the streaming updater was blocking on commits but updates never resumed. To be honest I was in a bit of a rush to meet a deadline so after spending a day or so tinkering I bailed out and just wrote a component by

Re: StreamingUpdateSolrServer hangs

2010-04-09 Thread Yonik Seeley
Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: * SOLR-1711: SolrJ - StreamingUpdateSolrServer had a race condition that could halt the streaming of documents.

StreamingUpdateSolrServer hangs

2010-02-05 Thread Stephen Meyer
I am trying to use the StreamingUpdateSolrServer to index a bunch of bibliographic data and it is hanging up every time I run it. Sometimes it hangs after about 100k records (after about 2 minutes), sometimes after 4M records (after about 80 minutes) and all different intervals in between. It