I have also just ran into this a few times over the weekend in a newly
deployed system. We are running Solr 4.0 Beta (not using SolrCloud) and it
is hosted via AWS.
I have a RabbitMQ consumer that reads updates from a queue and posts
updates to Solr via SolrJ. There is quite a bit of error handling around
the indexing request, and even if Solr is not live the consumer application
successfully logs the exception and attempts to move along in the queue.
There are two consumer applications running at once, and at times processes
400 requests per minute. The high volume times is not necessarily when this
problem occurs, though.
This exception is causing the entire application to hang - which is
surprising considering all SolrJ logic is wrapped with try/catches. Has
anyone found out more information regarding the possible keep alive bug?
Any insight is much appreciated.
Thanks,
Briggs Thompson
Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
tryExecute
INFO: I/O exception (java.net.SocketException) caught when processing
request: Broken pipe
Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
tryExecute
INFO: Retrying request
Oct 8, 2012 7:25:48 AM com.<>.rabbitmq.worker.SolrWriter work
SEVERE: {"id":4049703,"datetime":"2012-10-08 07:22:05"}
IOException occured when talking to server at:
<<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
server>
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at:
<<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
server>
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:362)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69)
at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:96)
at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:79)
at com.<>.solr.SolrIndexService.IndexCoupon(SolrIndexService.java:57)
at com.<>.solr.SolrIndexService.Index(SolrIndexService.java:36)
at com.<>.rabbitmq.worker.SolrWriter.work(SolrWriter.java:47)
at com.<>.rabbitmq.job.Runner.run(Runner.java:84)
at com.<>.rabbitmq.job.SolrConsumer.main(SolrConsumer.java:10)
Caused by: org.apache.http.client.ClientProtocolException
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:306)
... 10 more
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot
retry request with a non-repeatable request entity. The cause lists the
reason the original request failed.
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
... 13 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
at
org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154)
at
org.apache.http.impl.conn.LoggingSessionOutputBuffer.flush(LoggingSessionOutputBuffer.java:95)
at
org.apache.http.impl.io.ChunkedOutputStream.flush(ChunkedOutputStream.java:178)
at
org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:72)
at
org.apache.http.entity.mime.HttpMultipart.doWriteTo(HttpMultipart.java:206)
at org.apache.http.entity.mime.HttpMultipart.writeTo(HttpMultipart.java:224)
at
org.apache.http.entity.mime.MultipartEntity.writeTo(MultipartEntity.java:183)
at
org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
at
org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
at
org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
at
org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
at
org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227)
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712)
... 15 more
Oct 8, 2012 7:27:22 AM com.<>.rabbitmq.worker.SolrWriter work
SEVERE: {"id":4049704,"datetime":"2012-10-08 07:22:05"}
Server at
<<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
server> returned non ok status:500, message:Internal Server Error
org.apache.solr.client.solrj.SolrServerException: Server at <server> returned
non ok status:500, message:Internal Server Error
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:328)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69)
at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:96)
at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:79)
at com.<>.solr.SolrIndexService.IndexCoupon(SolrIndexService.java:57)
at com.<>.solr.SolrIndexService.Index(SolrIndexService.java:36)
at com.<>.rabbitmq.worker.SolrWriter.work(SolrWriter.java:47)
at com.<>.rabbitmq.job.Runner.run(Runner.java:84)
at com.<>.rabbitmq.job.SolrConsumer.main(SolrConsumer.java:10)
On Fri, Oct 5, 2012 at 4:20 PM, Erick Erickson <erickerick...@gmail.com>wrote:
> Well, using embedded Solr isn't necessarily indicated. I have a couple
> of questions.
> 1> you say 30 tps. Are you sending a single doc at a time or batching them
> up? I.e. server.add(doclist) or server.add(doc)?
>
> 2> Http isn't actually an inefficient protocol, I think the whole idea of
> using embedded Solr is a red herring if that's the concern.
>
> 3> bq: " call the VIP in front of the SOLR nodes to distribute the update
> load"
> What does that mean? Only one process should be writing to an index
> at a time. So there should be only one address to which all the
> updates go. The servlet container will stack up multiple incoming requests.
> So I totally don't understand what you're trying to accomplish here.
>
> 4> You haven't told us what version of Solr you're using, so the stack
> trace
> isn't all that helpful. We can pick random Solr versions and hope they
> match
> the line numbers, but that's not too efficient.....
>
> 5> Have you thought about just putting some fallback logic in your SolrJ
> program
> and having it sleep for a few seconds and retrying?
>
> Best
> Erick
>
> On Fri, Oct 5, 2012 at 4:35 PM, Sushil jain <jain.ayushm...@gmail.com>
> wrote:
> > If you need to use solr in an embedded application, this is the
> recommended
> > approach. It allows you to work with the same interface whether or not
> you
> > have access to HTTP.
> >
> > And it is not thread safe.
> >
> > On Sat, Oct 6, 2012 at 1:58 AM, balaji.gandhi
> > <balaji.gan...@apollogrp.edu>wrote:
> >
> >> Sushil, we are trying to call the VIP in front of the SOLR nodes to
> >> distribute the update load.
> >>
> >> Also is EmbeddedSolrServer thread safe?
> >>
> >> Balaji Gandhi, Senior Software Developer, Horizontal Platform Services
> >> Product Engineering │ Apollo Group, Inc.
> >> 1225 W. Washington St. | AZ23 | Tempe, AZ 85281
> >> Phone: 602.713.2417 | Email: balaji.gan...@apollogrp.edu<mailto:
> >> balaji.gan...@apollogrp.edu>
> >>
> >> P Go Green. Don't Print. Moreover soft copies can be indexed by
> algorithms.
> >>
> >> From: Sushil jain [via Lucene] [mailto:
> >> ml-node+s472066n4012179...@n3.nabble.com]
> >> Sent: Friday, October 05, 2012 1:23 PM
> >> To: Balaji Gandhi
> >> Subject: Re: SolrJ - IOException
> >>
> >> Yes, I'd recommend EmbeddedSolrServer, because it doesn't require any
> web
> >> server for read/write/update/delete operations.
> >>
> >> On Sat, Oct 6, 2012 at 1:48 AM, balaji.gandhi
> >> <[hidden email]</user/SendEmail.jtp?type=node&node=4012179&i=0>>wrote:
> >>
> >> > Sushil,
> >> >
> >> > 30 TPS = 30 transactions (updates) per second.
> >> >
> >> > Is the recommendation to use EmbeddedSolrServer instead of
> >> HttpSolrServer?
> >> >
> >> > Thanks,
> >> > Balaji
> >> >
> >> > Balaji Gandhi, Senior Software Developer, Horizontal Platform Services
> >> > Product Engineering │ Apollo Group, Inc.
> >> > 1225 W. Washington St. | AZ23 | Tempe, AZ 85281
> >> > Phone: 602.713.2417 | Email: [hidden
> >> email]</user/SendEmail.jtp?type=node&node=4012179&i=1><mailto:
> >> > [hidden email]</user/SendEmail.jtp?type=node&node=4012179&i=2>>
> >> >
> >> > P Go Green. Don't Print. Moreover soft copies can be indexed by
> >> algorithms.
> >> >
> >> > From: Sushil jain [via Lucene] [mailto:
> >> > [hidden email]</user/SendEmail.jtp?type=node&node=4012179&i=3>]
> >> > Sent: Friday, October 05, 2012 1:14 PM
> >> > To: Balaji Gandhi
> >> > Subject: Re: SolrJ - IOException
> >> >
> >> > Balaji,
> >> >
> >> > What is 30 TPS ?
> >> >
> >> > Toke,
> >> >
> >> > You should use EmbeddedSolrServer Instead.
> >> >
> >> >
> >> >
> >> > On Fri, Oct 5, 2012 at 11:42 PM, balaji.gandhi
> >> > <[hidden email]</user/SendEmail.jtp?type=node&node=4012172&i=0>>wrote:
> >> >
> >> > > Hi Toke,
> >> > >
> >> > > Were you able to find anything on this issue? We are running at 30
> TPS
> >> > and
> >> > > using the default HttpSolrServer for the posts.
> >> > >
> >> > > [cid:image001.png@01CDA2EA.370A6ED0]
> >> > >
> >> > > Thanks,
> >> > > Balaji
> >> > >
> >> > > Balaji Gandhi, Senior Software Developer, Horizontal Platform
> Services
> >> > > Product Engineering │ Apollo Group, Inc.
> >> > > 1225 W. Washington St. | AZ23 | Tempe, AZ 85281
> >> > > Phone: 602.713.2417 | Email: [hidden
> >> > email]</user/SendEmail.jtp?type=node&node=4012172&i=1><mailto:
> >> > > [hidden email]</user/SendEmail.jtp?type=node&node=4012172&i=2>>
> >> > >
> >> > > P Go Green. Don't Print. Moreover soft copies can be indexed by
> >> > algorithms.
> >> > >
> >> > > From: Toke Eskildsen [via Lucene] [mailto:
> >> > > [hidden email]</user/SendEmail.jtp?type=node&node=4012172&i=3>]
> >> > > Sent: Tuesday, September 25, 2012 12:19 AM
> >> > > To: Balaji Gandhi
> >> > > Subject: Re: SolrJ - IOException
> >> > >
> >> > > On Tue, 2012-09-25 at 01:50 +0200, balaji.gandhi wrote:
> >> > > > I am encountering this error randomly (under load) when posting to
> >> Solr
> >> > > > using SolrJ.
> >> > > >
> >> > > > Has anyone encountered a similar error?
> >> > > >
> >> > > > org.apache.solr.client.solrj.SolrServerException: IOException
> occured
> >> > > when
> >> > > > talking to server at: http://localhost:8080/solr/profile at
> >> > > >
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:414)
> >> > > [...]
> >> > >
> >> > > This looks suspiciously like a potential bug in the HTTP keep-alive
> >> flow
> >> > > that we encountered some weeks ago. I am guessing that you are
> issuing
> >> > > more than 100 separate updates/second. Could you please provide the
> >> full
> >> > > stack trace?
> >> > >
> >> > >
> >> > > ________________________________
> >> > > If you reply to this email, your message will be added to the
> >> discussion
> >> > > below:
> >> > >
> >> >
> >>
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4010082.html
> >> > > To unsubscribe from SolrJ - IOException, click here<
> >> > > >.
> >> > > NAML<
> >> > >
> >> >
> >>
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >> > > >
> >> > >
> >> > > ________________________________
> >> > > This message is private and confidential. If you have received it in
> >> > > error, please notify the sender and remove it from your system.
> >> > >
> >> > >
> >> > >
> >> > > image001.png (37K) <
> >> > >
> http://lucene.472066.n3.nabble.com/attachment/4012130/0/image001.png>
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > View this message in context:
> >> > >
> >> >
> >>
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012130.html
> >> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >
> >> > ________________________________
> >> > If you reply to this email, your message will be added to the
> discussion
> >> > below:
> >> >
> >>
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012172.html
> >> > To unsubscribe from SolrJ - IOException, click here<
> >> > >.
> >> > NAML<
> >> >
> >>
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >> > >
> >> >
> >> > ________________________________
> >> > This message is private and confidential. If you have received it in
> >> > error, please notify the sender and remove it from your system.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012176.html
> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the discussion
> >> below:
> >>
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012179.html
> >> To unsubscribe from SolrJ - IOException, click here<
> >>
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4010026&code=YmFsYWppLmdhbmRoaUBhcG9sbG9ncnAuZWR1fDQwMTAwMjZ8LTEwNzE2NTA1NDI=
> >> >.
> >> NAML<
> >>
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >> >
> >>
> >> ________________________________
> >> This message is private and confidential. If you have received it in
> >> error, please notify the sender and remove it from your system.
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012182.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
>
