Re: Solr JVM Memory settings

Hendrik Haddorp Mon, 15 Oct 2018 00:49:30 -0700

I wasn't stating that Docker is the solution at all. I was also statingthat the native memory would go down if you limit the heap.I'm running Solr in Docker with a memory limit and thus have to makesure that the memory is limited as otherwise Linux kills the JVM. Forthat I'm limiting the heap and meta space, which is also ofter referedas native memory. For the Oracle JVM (version 8) the heap and meta spacecan be limited. There are a few more memory areas, which I believe youcan not limit correctly. Just putting Solr in Docker will of course notdo any of that for you.


On 12.10.2018 19:59, Christopher Schultz wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Hendrik,

On 10/12/18 02:36, Hendrik Haddorp wrote:

Those constraints can be easily set if you are using Docker. The
problem is however that at least up to Oracle Java 8, and I believe
quite a bit further, the JVM is not at all aware about those
limits. That's why when running Solr in Docker you really need to
make sure that you set the memory limits lower. I usually set the
heap and metaspace size. How you set them depends again a bit on
your Solr configuration. I prefer the JVM to crash due to memory
limits rather then the Linux OOM Killer killing the JVM as the
OutOfMemoryError from the JVM does at least state what memory was
out.

Limiting the native memory used by attempting to limit the heap is not
actually limiting the native memory used. It's just an attempt to do
so. If you limit the native memory using OS limits (or, using Docker,
simply make it look like there is less system memory) then you haven't
actually achieved anything. You could have done that simply by
lowering heap values and avoided the complexity of Docker, etc.

- -chris

On 11.10.2018 16:45, Christopher Schultz wrote: Shawn,

On 10/11/18 12:54 AM, Shawn Heisey wrote:

On 10/10/2018 10:08 PM, Sourav Moitra wrote:

We have a Solr server with 8gb of memory. We are using solr
in cloud mode, solr version is 7.5, Java version is Oracle
Java 9 and settings for Xmx and Xms value is 2g but we are
observing that the RAM getting used to 98% when doing
indexing.

How can I ensure that SolrCloud doesn't use more than N GB
of memory ?

Where precisely are you seeing the 98% usage?  It is
completely normal for a modern operating system to report
that almost all the system memory is in use, at least after
the system has been shuffling a lot of data.  All modern
operating systems will use memory that has not been
specifically allocated to programs for disk caching purposes,
and system information tools will generally indicate that
this memory is in use, even though it can be instantly
claimed by any program that requests it.

https://en.wikipedia.org/wiki/Page_cache

If you tell a Java program that it is limited to a 2GB heap,
then that program will never use more than 2GB, plus a little
extra for the java runtime itself.  I cannot give you an
exact figure for that little bit extra.  But every bit of
data on disk that Solr accesses will end up (at least
temporarily) in the operating system's disk cache -- using
that unallocated memory.

https://wiki.apache.org/solr/SolrPerformanceProblems#RAM

To be fair, the JVM can use *much more* memory than you have
specified for your Java heap. It's just that the Java heap itself
wont exceed those values.

The JVM uses quite a bit of native memory which isn't counted in
the Java heap. There is only one way I know of to control that, and
it's to set a process-limit at the OS level on the amount of
memory allowed. I'm not sure how sensitive to those limits the JVM
actually is, so attempting to artificially constrain the JVM might
end up with a native OOM crash.

-chris

Re: Solr JVM Memory settings

Reply via email to