I tried to profile the memory of each solr node. I can see the GC activity
going higher as much as 98% and there are many instances where it has gone
up at 10+%. In one of the solr node I can see it going to 45%.
Memory is fully used and have gone to the maximum usage of heap which is
set to 24g. During other search I can see the error
*org.apache.solr.common.SolrException: no servers hosting shard.*
Few nodes are in gone state. There are many instances of
*org.apache.solr.common.SolrException:
org.apache.zookeeper.KeeperException$SessionExpiredException.*
GC logs shows a very busy garbage collection.Please provide your inputs.

On Tue, Aug 18, 2015 at 10:38 AM, Modassar Ather <modather1...@gmail.com>
wrote:

> Shawn! The container I am using is jetty only and the JVM setting I am
> using is the default one which comes with Solr startup scripts. Yes I have
> changed the JVM memory setting as mentioned.
> Kindly help me understand, even if there is a a GC pause why the solr node
> will go down. At least for other queries is should not throw exception of
> *org.apache.solr.common.SolrException: no servers hosting shard.*
> Why the node will throw above exception even a huge query is time out or
> may have taken lot of resources. Kindly help me understand in what
> conditions such exception can arise as I am not fully aware of it.
>
> Daniel! The error logs do not say if it was JVM crash or just solr. But by
> the exception I understand that it might have gone to a state from where it
> recovered after sometime. I did not restart the Solr.
>
> On Mon, Aug 17, 2015 at 10:12 PM, Daniel Collins <danwcoll...@gmail.com>
> wrote:
>
>> When you say "the solr node goes down", what do you mean by that? From
>> your
>> comment on the logs, you obviously lose the solr core at best (you do
>> realize only having a single replica is inherently susceptible to failure,
>> right?)
>> But do you mean the Solr Core drops out of the collection (ZK timeout),
>> the
>> JVM stops, the whole machine crashes?
>>
>> On 17 August 2015 at 14:17, Shawn Heisey <apa...@elyograg.org> wrote:
>>
>> > On 8/17/2015 5:45 AM, Modassar Ather wrote:
>> > > The servers have 32g memory each. Solr JVM memory is set to -Xms20g
>> > > -Xmx24g. There are no OOM in logs.
>> >
>> > Are you starting Solr 5.2.1 with the included start script, or have you
>> > installed it into another container?
>> >
>> > Assuming you're using the download's "bin/solr" script, that will
>> > normally set Xms and Xmx to the same value, so if you have overridden
>> > the memory settings such that you can have different values in Xms and
>> > Xmx, have you also overridden the garbage collection parameters?  If you
>> > have, what are they set to now?  You can see all arguments used on
>> > startup in the "JVM" section of the admin UI dashboard.
>> >
>> > If you've installed in an entirely different container, or you have
>> > overridden the garbage collection settings, then a 24GB heap might have
>> > extreme garbage collection pauses, lasting long enough to exceed the
>> > timeout.
>> >
>> > Giving 24 out of 32GB to Solr will mean that there is only (at most) 8GB
>> > left over for caching the index.  With 200GB of index, this is nowhere
>> > near enough, and is another likely source of Solr performance problems
>> > that cause timeouts.  This is what Upayavira was referring to in his
>> > reply.  For good performance with 200GB of index, you may need a lot
>> > more than 32GB of total RAM.
>> >
>> > https://wiki.apache.org/solr/SolrPerformanceProblems
>> >
>> > This wiki page also describes how you can use jconsole to judge how much
>> > heap you actually need.  24GB may be too much.
>> >
>> > Thanks,
>> > Shawn
>> >
>> >
>>
>
>

Reply via email to