Those constraints can be easily set if you are using Docker. The problem
is however that at least up to Oracle Java 8, and I believe quite a bit
further, the JVM is not at all aware about those limits. That's why when
running Solr in Docker you really need to make sure that you set the
memory limits lower. I usually set the heap and metaspace size. How you
set them depends again a bit on your Solr configuration. I prefer the
JVM to crash due to memory limits rather then the Linux OOM Killer
killing the JVM as the OutOfMemoryError from the JVM does at least state
what memory was out.
Hendrik
On 11.10.2018 16:45, Christopher Schultz wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Shawn,
On 10/11/18 12:54 AM, Shawn Heisey wrote:
On 10/10/2018 10:08 PM, Sourav Moitra wrote:
We have a Solr server with 8gb of memory. We are using solr in
cloud mode, solr version is 7.5, Java version is Oracle Java 9
and settings for Xmx and Xms value is 2g but we are observing
that the RAM getting used to 98% when doing indexing.
How can I ensure that SolrCloud doesn't use more than N GB of
memory ?
Where precisely are you seeing the 98% usage? It is completely
normal for a modern operating system to report that almost all the
system memory is in use, at least after the system has been
shuffling a lot of data. All modern operating systems will use
memory that has not been specifically allocated to programs for
disk caching purposes, and system information tools will generally
indicate that this memory is in use, even though it can be
instantly claimed by any program that requests it.
https://en.wikipedia.org/wiki/Page_cache
If you tell a Java program that it is limited to a 2GB heap, then
that program will never use more than 2GB, plus a little extra for
the java runtime itself. I cannot give you an exact figure for
that little bit extra. But every bit of data on disk that Solr
accesses will end up (at least temporarily) in the operating
system's disk cache -- using that unallocated memory.
https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
To be fair, the JVM can use *much more* memory than you have specified
for your Java heap. It's just that the Java heap itself wont exceed
those values.
The JVM uses quite a bit of native memory which isn't counted in the
Java heap. There is only one way I know of to control that, and it's
to set a process-limit at the OS level on the amount of memory
allowed. I'm not sure how sensitive to those limits the JVM actually
is, so attempting to artificially constrain the JVM might end up with
a native OOM crash.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlu/YgoACgkQHPApP6U8
pFjcbBAAgYegk20pYvfu3vcrAKxj3s+RSMGRPZ+nN5g0KYQFuhFgptYE+TqjLfBX
geekQUNqNUHO5psMA5q/6m6b3LwpqrMxJiapv0wWQ2wPah21CgLs/P/iG+elNQ63
H0ZXbe3wX0P0onZbP4+sfDyzhujZ+5+gMooK87o8Q4z91hIVX1EZfM4lcaZ3pbnb
JJ44YorWGPpXjQNEtOHfS7l/Q+8+6+XfEyfKha3JpRFcwcqgLpv23Koy4xgxgYr+
PMqfjptMBMjZ04xSdd491crm2yZowv3KH1Ss8v/L51rknGYPxCEkdKvPrUlpn+Rb
4WnQS6H//dJvQaLum/qR9Jxd+3vc13K7Mn++5Lu+jMbeEgaJU2hD4/ap/KMtFCqn
eIXl6HQYPW36sVcm/MIpkRvAgx8vri17sd3/5sOYaETrp4SMxMN5W44GvgDdkbGF
R9/tVBCFWb3p+o8eSKUf7QmARiN69DHGVwtQHWMIp8K9893IeHUNgVXKD7281zLB
AjHPc7QTvAn4xne0X9lvQjr+YKOPxd9FFqMBejdKht9aBFQvApma9LtJT3FInrob
QkSIx594KhoRltRy7E9t3XuWWGg8ujiuzKl6SEPsgXUC2Opwr4Wwu1yn9dCWkFJz
RzCKbaDBaNmrK6HSEsoNvS+yQPksPxM8MuchFaCAMZpVOsobCM0=
=77dD
-----END PGP SIGNATURE-----
