I don't think you gain anything on the Solr end of things by using multiple threads if you are already using StreamingUpdateSolrServer. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ---- > From: vivek sar <vivex...@gmail.com> > To: solr-user@lucene.apache.org > Sent: Thursday, April 9, 2009 5:47:11 PM > Subject: Re: httpclient.ProtocolException using Solrj > > Here is what I'm doing, > > SolrServer server = new StreamingUpdateSolrServer(url, 1000,5); > > server.addBeans(dataList); //where dataList is Listwith 10K elements > > I run two threads each using the same server object and then each call > server.addBeans(...). > > I'm able to get 50K/sec inserted using that, but the commit after that > (after 100k records) takes 70sec - which messes up the avg time. > > There are two problems here, > > 1) Once in a while I get "connection reset" error, > > Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:168) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at > org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) > > Note: if I use CommonsHttpSolrServer I get the buffer error. > > 2) The commit takes way too long for every 100k (I may commit more > often if this can not be improved) > > I'm trying to fix this error problem which happens only if I run two > threads both calling addBeans (10k at a time). One thread work fine. > I'm not sure how can I use the MultiThreadedConnectionManager to > create StreamingUpdateSolrServer and if they would help? > > Thanks, > -vivek > > 2009/4/9 Noble Paul നോബിള് नोब्ळ् : > > using a single request is the fatest > > > > > http://wiki.apache.org/solr/Solrj#head-2046bbaba3759b6efd0e33e93f5502038c01ac65 > > > > I could index at the rate of 10,000 docs/sec using this and > BinaryRequestWriter > > > > On Thu, Apr 9, 2009 at 10:36 PM, vivek sar wrote: > >> I'm inserting 10K in a batch (using addBeans method). I read somewhere > >> in the wiki that it's better to use the same instance of SolrServer > >> for better performance. Would MultiThreadedConnectionManager help? How > >> do I use it? > >> > >> I also wanted to know how can use EmbeddedSolrServer - does my app > >> needs to be running in the same jvm with Solr webapp? > >> > >> Thanks, > >> -vivek > >> > >> 2009/4/9 Noble Paul നോബിള് नोब्ळ् : > >>> how many documents are you inserting ? > >>> may be you can create multiple instances of CommonshttpSolrServer and > >>> upload in parallel > >>> > >>> > >>> On Thu, Apr 9, 2009 at 11:58 AM, vivek sar wrote: > >>>> Thanks Shalin and Paul. > >>>> > >>>> I'm not using MultipartRequest. I do share the same SolrServer between > >>>> two threads. I'm not using MultiThreadedHttpConnectionManager. I'm > >>>> simply using CommonsHttpSolrServer to create the SolrServer. I've also > >>>> tried StreamingUpdateSolrServer, which works much faster, but does > >>>> throws "connection reset" exception once in a while. > >>>> > >>>> Do I need to use MultiThreadedHttpConnectionManager? I couldn't find > >>>> anything on it on Wiki. > >>>> > >>>> I was also thinking of using EmbeddedSolrServer - in what case would I > >>>> be able to use it? Does my application and the Solr web app need to > >>>> run into the same JVM for this to work? How would I use the > >>>> EmbeddedSolrServer? > >>>> > >>>> Thanks, > >>>> -vivek > >>>> > >>>> > >>>> On Wed, Apr 8, 2009 at 10:46 PM, Shalin Shekhar Mangar > >>>> wrote: > >>>>> Vivek, do you share the same SolrServer instance between your two > >>>>> threads? > >>>>> If so, are you using the MultiThreadedHttpConnectionManager when > >>>>> creating > >>>>> the HttpClient instance? > >>>>> > >>>>> On Wed, Apr 8, 2009 at 10:13 PM, vivek sar wrote: > >>>>> > >>>>>> single thread everything works fine. Two threads are fine too for a > >>>>>> while and all the sudden problem starts happening. > >>>>>> > >>>>>> I tried indexing using REST services as well (instead of Solrj), but > >>>>>> with that too I get following error after a while, > >>>>>> > >>>>>> 2009-04-08 10:04:08,126 ERROR [indexerThreadPool-5] Indexer - > >>>>>> indexData()-> Failed to index > >>>>>> java.net.SocketException: Broken pipe > >>>>>> at java.net.SocketOutputStream.socketWrite0(Native Method) > >>>>>> at > >>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > >>>>>> at > >>>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:136) > >>>>>> at > java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) > >>>>>> at java.io.FilterOutputStream.write(FilterOutputStream.java:80) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.methods.StringRequestEntity.writeRequest(StringRequestEntity.java:145) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > >>>>>> at > >>>>>> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > >>>>>> > >>>>>> > >>>>>> Note, I'm using "simple" lock type. I'd tried "single" type before > >>>>>> that once caused index corruption so I switched to "simple". > >>>>>> > >>>>>> Thanks, > >>>>>> -vivek > >>>>>> > >>>>>> 2009/4/8 Noble Paul നോബിള് नोब्ळ् : > >>>>>> > do you see the same problem when you use a single thread? > >>>>>> > > >>>>>> > what is the version of SolrJ that you use? > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > On Wed, Apr 8, 2009 at 1:19 PM, vivek sar wrote: > >>>>>> >> Hi, > >>>>>> >> > >>>>>> >> Any ideas on this issue? I ran into this again - once it starts > >>>>>> >> happening it keeps happening. One of the thread keeps failing. Here > >>>>>> >> are my SolrServer settings, > >>>>>> >> > >>>>>> >> int socketTO = 0; > >>>>>> >> int connectionTO = 100; > >>>>>> >> int maxConnectionPerHost = 10; > >>>>>> >> int maxTotalConnection = 50; > >>>>>> >> boolean followRedirects = false; > >>>>>> >> boolean allowCompression = true; > >>>>>> >> int maxRetries = 1; > >>>>>> >> > >>>>>> >> Note, I'm using two threads to simultaneously write to the same > >>>>>> >> index. > >>>>>> >> > >>>>>> >> org.apache.solr.client.solrj.SolrServerException: > >>>>>> >> org.apache.commons.httpclient.ProtocolException: Unbuffered entity > >>>>>> >> enclosing request can not be repeated. > >>>>>> >> at > >>>>>> > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:470) > >>>>>> >> at > >>>>>> > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242) > >>>>>> >> at > >>>>>> > org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259) > >>>>>> >> at > >>>>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48) > >>>>>> >> at > >>>>>> org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:57) > >>>>>> >> > >>>>>> >> Thanks, > >>>>>> >> -vivek > >>>>>> >> > >>>>>> >> On Sat, Apr 4, 2009 at 1:07 AM, vivek sar wrote: > >>>>>> >>> Hi, > >>>>>> >>> > >>>>>> >>> I'm sending 15K records at once using Solrj (server.addBeans(...)) > >>>>>> >>> and have two threads writing to same index. One thread goes fine, > >>>>>> >>> but > >>>>>> >>> the second thread always fails with, > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> org.apache.solr.client.solrj.SolrServerException: > >>>>>> >>> org.apache.commons.httpclient.ProtocolException: Unbuffered entity > >>>>>> >>> enclosing request can not be repeated. > >>>>>> >>> at > >>>>>> > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:470) > >>>>>> >>> at > >>>>>> > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242) > >>>>>> >>> at > >>>>>> > org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259) > >>>>>> >>> at > >>>>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48) > >>>>>> >>> at > >>>>>> org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:57) > >>>>>> >>> at > >>>>>> > com.apple.afterchat.indexer.solr.handler.BeanIndexHandler.indexData(BeanIndexHandler.java:44) > >>>>>> >>> at > >>>>>> com.apple.afterchat.indexer.Indexer.indexData(Indexer.java:77) > >>>>>> >>> at com.apple.afterchat.indexer.Indexer.run(Indexer.java:39) > >>>>>> >>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) > >>>>>> >>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > >>>>>> >>> at java.lang.Thread.run(Thread.java:637) > >>>>>> >>> Caused by: org.apache.commons.httpclient.ProtocolException: > Unbuffered > >>>>>> >>> entity enclosing request can not be repeated. > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:487) > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114) > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096) > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > >>>>>> >>> at > >>>>>> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > >>>>>> >>> at > >>>>>> > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:417) > >>>>>> >>> > >>>>>> >>> Does anyone know what could be the problem? > >>>>>> >>> > >>>>>> >>> Thanks, > >>>>>> >>> -vivek > >>>>>> >>> > >>>>>> >> > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > -- > >>>>>> > --Noble Paul > >>>>>> > > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Regards, > >>>>> Shalin Shekhar Mangar. > >>>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> --Noble Paul > >>> > >> > > > > > > > > -- > > --Noble Paul > >
