Sathya,
We're still missing a fair amount of information here though it looks like your
cluster is healthy. How are you indexing and what's the request you're sending
that results in the error you're seeing? Have you checked your nodes' logs for
errors that correspond with the one you're seeing while indexing?
Thanks,
Greg
On Mar 22, 2014, at 2:32 PM, Shalin Shekhar Mangar <shalinman...@gmail.com>
wrote:
> Thanks Michael! I just committed your fix. It will be released with 4.7.1
>
> On Fri, Mar 21, 2014 at 8:30 PM, Michael Sokolov
> <msoko...@safaribooksonline.com> wrote:
>> I just managed to track this down -- as you said the disconnect was a red
>> herring.
>>
>> Ultimately the problem was caused by a custom analysis component we wrote
>> that was raising an IOException -- it was missing some configuration files
>> it relies on.
>>
>> What might be interesting for solr devs to have a look at is that exception
>> was completely swallowed by JavabinCodec, making it very difficult to track
>> down the problem. Furthermore -- if the /add request was routed directly to
>> the shard where the document was destined to end up, then the IOException
>> raised by the analysis component (a char filter) showed up in the Solr HTTP
>> response (probably because my client used XML format in one test -- javabin
>> is used internally in SolrCloud). But if the request was routed to a
>> different shard, then the only exception that showed up anywhere (in the
>> logs, in the HTTP response) was kind of irrelevant.
>>
>> I think this could be fixed pretty easily; see SOLR-5985 for my suggestion.
>>
>> -Mike
>>
>>
>>
>> On 03/21/2014 10:20 AM, Greg Walters wrote:
>>>
>>> Broken pipe errors are generally caused by unexpected disconnections and
>>> are some times hard to track down. Given the stack traces you've provided
>>> it's hard to point to any one thing and I suspect the relevant information
>>> was snipped out in the "long dump of document fields". You might grab the
>>> entire error from the client you're uploading documents with, the server
>>> you're connected to and any other nodes that have an error at the same time
>>> and put it on pastebin or the like.
>>>
>>> Thanks,
>>> Greg
>>>
>>> On Mar 20, 2014, at 3:36 PM, Michael Sokolov
>>> <msoko...@safaribooksonline.com> wrote:
>>>
>>>> I'm getting a similar exception when writing documents (on the client
>>>> side). I can write one document fine, but the second (which is being
>>>> routed
>>>> to a different shard) generates the error. It happens every time -
>>>> definitely not a resource issue or timing problem since this database is
>>>> completely empty -- I'm just getting started and running some tests, so
>>>> there must be some kind of setup problem. But it's difficult to diagnose
>>>> (for me, anyway)! I'd appreciate any insight, hints, guesses, etc. since
>>>> I'm stuck. Thanks!
>>>>
>>>> One node (the leader?) is reporting "Internal Server Error" in its log,
>>>> and another node (presumably the shard where the document is being
>>>> directed)
>>>> bombs out like this:
>>>>
>>>> ERROR - 2014-03-20 15:56:53.022; org.apache.solr.common.SolrException;
>>>> null:org.apache.solr.common.SolrException: ERROR adding document
>>>> SolrInputDocument(
>>>>
>>>> ... long dump of document fields
>>>>
>>>> )
>>>> at
>>>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:99)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>>>> at
>>>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
>>>> at
>>>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
>>>> at
>>>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
>>>> at
>>>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
>>>> at
>>>> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
>>>> at
>>>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>>>> at
>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>>> at
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
>>>> at
>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
>>>> ...
>>>> Caused by: java.net.SocketException: Broken pipe
>>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>>> at
>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
>>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
>>>> at
>>>> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
>>>> at
>>>> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
>>>> at
>>>> org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
>>>> at
>>>> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
>>>> at
>>>> org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119)
>>>> at
>>>> org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
>>>> at org.apache.coyote.Response.doWrite(Response.java:520)
>>>> at
>>>> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
>>>> ... 37 more
>>>>
>>>> This is with Solr 4.6.1, Tomcat 7. Here's my clusterstate.json. Updates
>>>> are being sent to the test1x3 collection
>>>>
>>>>
>>>> {
>>>> "test3x1":{
>>>> "shards":{
>>>> "shard1":{
>>>> "range":"80000000-d554ffff",
>>>> "state":"active",
>>>> "replicas":{"core_node1":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.37:8080/solr";,
>>>> "core":"test3x1_shard1_replica1",
>>>> "node_name":"10.4.24.37:8080_solr",
>>>> "leader":"true"}}},
>>>> "shard2":{
>>>> "range":"d5550000-2aa9ffff",
>>>> "state":"active",
>>>> "replicas":{"core_node3":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.39:8080/solr";,
>>>> "core":"test3x1_shard2_replica1",
>>>> "node_name":"10.4.24.39:8080_solr",
>>>> "leader":"true"}}},
>>>> "shard3":{
>>>> "range":"2aaa0000-7fffffff",
>>>> "state":"active",
>>>> "replicas":{"core_node2":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.38:8080/solr";,
>>>> "core":"test3x1_shard3_replica1",
>>>> "node_name":"10.4.24.38:8080_solr",
>>>> "leader":"true"}}}},
>>>> "maxShardsPerNode":"1",
>>>> "router":{"name":"compositeId"},
>>>> "replicationFactor":"1"},
>>>> "test1x3":{
>>>> "shards":{"shard1":{
>>>> "range":"80000000-7fffffff",
>>>> "state":"active",
>>>> "replicas":{
>>>> "core_node1":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.39:8080/solr";,
>>>> "core":"test1x3_shard1_replica2",
>>>> "node_name":"10.4.24.39:8080_solr",
>>>> "leader":"true"},
>>>> "core_node2":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.38:8080/solr";,
>>>> "core":"test1x3_shard1_replica1",
>>>> "node_name":"10.4.24.38:8080_solr"},
>>>> "core_node3":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.37:8080/solr";,
>>>> "core":"test1x3_shard1_replica3",
>>>> "node_name":"10.4.24.37:8080_solr"}}}},
>>>> "maxShardsPerNode":"1",
>>>> "router":{"name":"compositeId"},
>>>> "replicationFactor":"3"},
>>>> "test2x2":{
>>>> "shards":{
>>>> "shard1":{
>>>> "range":"80000000-ffffffff",
>>>> "state":"active",
>>>> "replicas":{
>>>> "core_node1":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.39:8080/solr";,
>>>> "core":"test2x2_shard1_replica1",
>>>> "node_name":"10.4.24.39:8080_solr"},
>>>> "core_node4":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.38:8080/solr";,
>>>> "core":"test2x2_shard1_replica2",
>>>> "node_name":"10.4.24.38:8080_solr",
>>>> "leader":"true"}}},
>>>> "shard2":{
>>>> "range":"0-7fffffff",
>>>> "state":"active",
>>>> "replicas":{
>>>> "core_node2":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.37:8080/solr";,
>>>> "core":"test2x2_shard2_replica1",
>>>> "node_name":"10.4.24.37:8080_solr",
>>>> "leader":"true"},
>>>> "core_node3":{
>>>> "state":"active",
>>>> "base_url":"http://10.4.24.39:8080/solr";,
>>>> "core":"test2x2_shard2_replica2",
>>>> "node_name":"10.4.24.39:8080_solr"}}}},
>>>> "maxShardsPerNode":"2",
>>>> "router":{"name":"compositeId"},
>>>> "replicationFactor":"2"}}
>>>>
>>>>
>>>>
>>>> On 03/20/2014 09:44 AM, Greg Walters wrote:
>>>>>
>>>>> Sathya,
>>>>>
>>>>> I assume you're using Solr Cloud. Please provide your clusterstate.json
>>>>> while you're seeing this issue and check your logs for any exceptions.
>>>>> With
>>>>> no information from you it's hard to troubleshoot any issues!
>>>>>
>>>>> Thanks,
>>>>> Greg
>>>>>
>>>>> On Mar 20, 2014, at 12:44 AM, Sathya <sathia.blacks...@gmail.com> wrote:
>>>>>
>>>>>> Hi Friends,
>>>>>>
>>>>>> I am new to Solr. I have 5 solr node in 5 different machine. When i
>>>>>> index
>>>>>> the data, sometimes "*No live SolrServers available to handle this
>>>>>> request*"
>>>>>> exception occur in 1 or 2 machines.
>>>>>>
>>>>>> I dont know why its happen and how to solve this. Kindly help me to
>>>>>> solve
>>>>>> this issue.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://lucene.472066.n3.nabble.com/Solr4-7-No-live-SolrServers-available-to-handle-this-request-tp4125679.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
