Re: Solr JVM Memory settings
I wasn't stating that Docker is the solution at all. I was also stating that the native memory would go down if you limit the heap. I'm running Solr in Docker with a memory limit and thus have to make sure that the memory is limited as otherwise Linux kills the JVM. For that I'm limiting the heap and meta space, which is also ofter refered as native memory. For the Oracle JVM (version 8) the heap and meta space can be limited. There are a few more memory areas, which I believe you can not limit correctly. Just putting Solr in Docker will of course not do any of that for you. On 12.10.2018 19:59, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hendrik, On 10/12/18 02:36, Hendrik Haddorp wrote: Those constraints can be easily set if you are using Docker. The problem is however that at least up to Oracle Java 8, and I believe quite a bit further, the JVM is not at all aware about those limits. That's why when running Solr in Docker you really need to make sure that you set the memory limits lower. I usually set the heap and metaspace size. How you set them depends again a bit on your Solr configuration. I prefer the JVM to crash due to memory limits rather then the Linux OOM Killer killing the JVM as the OutOfMemoryError from the JVM does at least state what memory was out. Limiting the native memory used by attempting to limit the heap is not actually limiting the native memory used. It's just an attempt to do so. If you limit the native memory using OS limits (or, using Docker, simply make it look like there is less system memory) then you haven't actually achieved anything. You could have done that simply by lowering heap values and avoided the complexity of Docker, etc. - -chris On 11.10.2018 16:45, Christopher Schultz wrote: Shawn, On 10/11/18 12:54 AM, Shawn Heisey wrote: On 10/10/2018 10:08 PM, Sourav Moitra wrote: We have a Solr server with 8gb of memory. We are using solr in cloud mode, solr version is 7.5, Java version is Oracle Java 9 and settings for Xmx and Xms value is 2g but we are observing that the RAM getting used to 98% when doing indexing. How can I ensure that SolrCloud doesn't use more than N GB of memory ? Where precisely are you seeing the 98% usage? It is completely normal for a modern operating system to report that almost all the system memory is in use, at least after the system has been shuffling a lot of data. All modern operating systems will use memory that has not been specifically allocated to programs for disk caching purposes, and system information tools will generally indicate that this memory is in use, even though it can be instantly claimed by any program that requests it. https://en.wikipedia.org/wiki/Page_cache If you tell a Java program that it is limited to a 2GB heap, then that program will never use more than 2GB, plus a little extra for the java runtime itself. I cannot give you an exact figure for that little bit extra. But every bit of data on disk that Solr accesses will end up (at least temporarily) in the operating system's disk cache -- using that unallocated memory. https://wiki.apache.org/solr/SolrPerformanceProblems#RAM To be fair, the JVM can use *much more* memory than you have specified for your Java heap. It's just that the Java heap itself wont exceed those values. The JVM uses quite a bit of native memory which isn't counted in the Java heap. There is only one way I know of to control that, and it's to set a process-limit at the OS level on the amount of memory allowed. I'm not sure how sensitive to those limits the JVM actually is, so attempting to artificially constrain the JVM might end up with a native OOM crash. -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvA4O0ACgkQHPApP6U8 pFiCHg/+P+/yoSrvMd2uMyDK16nMCOIdxAL1gdS++DqS+qPmch1BHJTA9nuHybF4 j6WElpCI7Q3HP/sgsGE8kHE6Kg+DFJNz7mGJqgXjnSkm90LzETRFMqa959fTgBo6 SILD4n4LnZI844VoaKb2gIVibr804hloxX5UDe0XYFp3EtcVi4QMC5Q2ovn8+RoJ S/LJx/VQi3AqtcCaEYAAKpYrKxO3OkoIKnN+oC55ag/16zh9StT2TUI03bBslcxn PkS5zdsSmsS7NydSR4Gn4C7wAGyL3hGoU6pD+GhvYE9EF29KxHXFSIe2FJQ6mdRf ikZvm17U8OFNwqlB4OOLziGvOkcmIgtqchnhUm80Qwtn0ZMbql2zwlIhOSPWbuPL lq3F09p1QBqPjbxJdrcmpoSFH8jvmIPdrPOl3BbPEmDzNdnF03sEGP5gDyJ9/INB AD/QhqvQEKUtMBPX+1/9dxOm+JyUDlARZQ7p4k1BeFjl2BI8imLUK/c6JlWJ757G QWk+0Ff3R02va+ITWNvGs5C1uOnu2g58eqAggREPWXmXAj9nqJ5EyPkNAaGJBheo NasGNSXVnjN+hk4QlMTAJ3C5u0Q5lW3HCOXj8Mufo7LE8M96OjRkM09o87NG9sGT EdX7V8Ypw758Jt9xcms6U9tC2TqekJ9AYu+VLsoGa4OZgy5hfDk= =Sq+f -END PGP SIGNATURE-
Re: Solr JVM Memory settings
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hendrik, On 10/12/18 02:36, Hendrik Haddorp wrote: > Those constraints can be easily set if you are using Docker. The > problem is however that at least up to Oracle Java 8, and I believe > quite a bit further, the JVM is not at all aware about those > limits. That's why when running Solr in Docker you really need to > make sure that you set the memory limits lower. I usually set the > heap and metaspace size. How you set them depends again a bit on > your Solr configuration. I prefer the JVM to crash due to memory > limits rather then the Linux OOM Killer killing the JVM as the > OutOfMemoryError from the JVM does at least state what memory was > out. Limiting the native memory used by attempting to limit the heap is not actually limiting the native memory used. It's just an attempt to do so. If you limit the native memory using OS limits (or, using Docker, simply make it look like there is less system memory) then you haven't actually achieved anything. You could have done that simply by lowering heap values and avoided the complexity of Docker, etc. - -chris > On 11.10.2018 16:45, Christopher Schultz wrote: Shawn, > > On 10/11/18 12:54 AM, Shawn Heisey wrote: On 10/10/2018 10:08 PM, Sourav Moitra wrote: > We have a Solr server with 8gb of memory. We are using solr > in cloud mode, solr version is 7.5, Java version is Oracle > Java 9 and settings for Xmx and Xms value is 2g but we are > observing that the RAM getting used to 98% when doing > indexing. > > How can I ensure that SolrCloud doesn't use more than N GB > of memory ? Where precisely are you seeing the 98% usage? It is completely normal for a modern operating system to report that almost all the system memory is in use, at least after the system has been shuffling a lot of data. All modern operating systems will use memory that has not been specifically allocated to programs for disk caching purposes, and system information tools will generally indicate that this memory is in use, even though it can be instantly claimed by any program that requests it. https://en.wikipedia.org/wiki/Page_cache If you tell a Java program that it is limited to a 2GB heap, then that program will never use more than 2GB, plus a little extra for the java runtime itself. I cannot give you an exact figure for that little bit extra. But every bit of data on disk that Solr accesses will end up (at least temporarily) in the operating system's disk cache -- using that unallocated memory. https://wiki.apache.org/solr/SolrPerformanceProblems#RAM > To be fair, the JVM can use *much more* memory than you have > specified for your Java heap. It's just that the Java heap itself > wont exceed those values. > > The JVM uses quite a bit of native memory which isn't counted in > the Java heap. There is only one way I know of to control that, and > it's to set a process-limit at the OS level on the amount of > memory allowed. I'm not sure how sensitive to those limits the JVM > actually is, so attempting to artificially constrain the JVM might > end up with a native OOM crash. > > -chris > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvA4O0ACgkQHPApP6U8 pFiCHg/+P+/yoSrvMd2uMyDK16nMCOIdxAL1gdS++DqS+qPmch1BHJTA9nuHybF4 j6WElpCI7Q3HP/sgsGE8kHE6Kg+DFJNz7mGJqgXjnSkm90LzETRFMqa959fTgBo6 SILD4n4LnZI844VoaKb2gIVibr804hloxX5UDe0XYFp3EtcVi4QMC5Q2ovn8+RoJ S/LJx/VQi3AqtcCaEYAAKpYrKxO3OkoIKnN+oC55ag/16zh9StT2TUI03bBslcxn PkS5zdsSmsS7NydSR4Gn4C7wAGyL3hGoU6pD+GhvYE9EF29KxHXFSIe2FJQ6mdRf ikZvm17U8OFNwqlB4OOLziGvOkcmIgtqchnhUm80Qwtn0ZMbql2zwlIhOSPWbuPL lq3F09p1QBqPjbxJdrcmpoSFH8jvmIPdrPOl3BbPEmDzNdnF03sEGP5gDyJ9/INB AD/QhqvQEKUtMBPX+1/9dxOm+JyUDlARZQ7p4k1BeFjl2BI8imLUK/c6JlWJ757G QWk+0Ff3R02va+ITWNvGs5C1uOnu2g58eqAggREPWXmXAj9nqJ5EyPkNAaGJBheo NasGNSXVnjN+hk4QlMTAJ3C5u0Q5lW3HCOXj8Mufo7LE8M96OjRkM09o87NG9sGT EdX7V8Ypw758Jt9xcms6U9tC2TqekJ9AYu+VLsoGa4OZgy5hfDk= =Sq+f -END PGP SIGNATURE-
Re: Solr JVM Memory settings
Those constraints can be easily set if you are using Docker. The problem is however that at least up to Oracle Java 8, and I believe quite a bit further, the JVM is not at all aware about those limits. That's why when running Solr in Docker you really need to make sure that you set the memory limits lower. I usually set the heap and metaspace size. How you set them depends again a bit on your Solr configuration. I prefer the JVM to crash due to memory limits rather then the Linux OOM Killer killing the JVM as the OutOfMemoryError from the JVM does at least state what memory was out. Hendrik On 11.10.2018 16:45, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shawn, On 10/11/18 12:54 AM, Shawn Heisey wrote: On 10/10/2018 10:08 PM, Sourav Moitra wrote: We have a Solr server with 8gb of memory. We are using solr in cloud mode, solr version is 7.5, Java version is Oracle Java 9 and settings for Xmx and Xms value is 2g but we are observing that the RAM getting used to 98% when doing indexing. How can I ensure that SolrCloud doesn't use more than N GB of memory ? Where precisely are you seeing the 98% usage? It is completely normal for a modern operating system to report that almost all the system memory is in use, at least after the system has been shuffling a lot of data. All modern operating systems will use memory that has not been specifically allocated to programs for disk caching purposes, and system information tools will generally indicate that this memory is in use, even though it can be instantly claimed by any program that requests it. https://en.wikipedia.org/wiki/Page_cache If you tell a Java program that it is limited to a 2GB heap, then that program will never use more than 2GB, plus a little extra for the java runtime itself. I cannot give you an exact figure for that little bit extra. But every bit of data on disk that Solr accesses will end up (at least temporarily) in the operating system's disk cache -- using that unallocated memory. https://wiki.apache.org/solr/SolrPerformanceProblems#RAM To be fair, the JVM can use *much more* memory than you have specified for your Java heap. It's just that the Java heap itself wont exceed those values. The JVM uses quite a bit of native memory which isn't counted in the Java heap. There is only one way I know of to control that, and it's to set a process-limit at the OS level on the amount of memory allowed. I'm not sure how sensitive to those limits the JVM actually is, so attempting to artificially constrain the JVM might end up with a native OOM crash. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlu/YgoACgkQHPApP6U8 pFjcbBAAgYegk20pYvfu3vcrAKxj3s+RSMGRPZ+nN5g0KYQFuhFgptYE+TqjLfBX geekQUNqNUHO5psMA5q/6m6b3LwpqrMxJiapv0wWQ2wPah21CgLs/P/iG+elNQ63 H0ZXbe3wX0P0onZbP4+sfDyzhujZ+5+gMooK87o8Q4z91hIVX1EZfM4lcaZ3pbnb JJ44YorWGPpXjQNEtOHfS7l/Q+8+6+XfEyfKha3JpRFcwcqgLpv23Koy4xgxgYr+ PMqfjptMBMjZ04xSdd491crm2yZowv3KH1Ss8v/L51rknGYPxCEkdKvPrUlpn+Rb 4WnQS6H//dJvQaLum/qR9Jxd+3vc13K7Mn++5Lu+jMbeEgaJU2hD4/ap/KMtFCqn eIXl6HQYPW36sVcm/MIpkRvAgx8vri17sd3/5sOYaETrp4SMxMN5W44GvgDdkbGF R9/tVBCFWb3p+o8eSKUf7QmARiN69DHGVwtQHWMIp8K9893IeHUNgVXKD7281zLB AjHPc7QTvAn4xne0X9lvQjr+YKOPxd9FFqMBejdKht9aBFQvApma9LtJT3FInrob QkSIx594KhoRltRy7E9t3XuWWGg8ujiuzKl6SEPsgXUC2Opwr4Wwu1yn9dCWkFJz RzCKbaDBaNmrK6HSEsoNvS+yQPksPxM8MuchFaCAMZpVOsobCM0= =77dD -END PGP SIGNATURE-
Re: Solr JVM Memory settings
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shawn, On 10/11/18 12:54 AM, Shawn Heisey wrote: > On 10/10/2018 10:08 PM, Sourav Moitra wrote: >> We have a Solr server with 8gb of memory. We are using solr in >> cloud mode, solr version is 7.5, Java version is Oracle Java 9 >> and settings for Xmx and Xms value is 2g but we are observing >> that the RAM getting used to 98% when doing indexing. >> >> How can I ensure that SolrCloud doesn't use more than N GB of >> memory ? > > Where precisely are you seeing the 98% usage? It is completely > normal for a modern operating system to report that almost all the > system memory is in use, at least after the system has been > shuffling a lot of data. All modern operating systems will use > memory that has not been specifically allocated to programs for > disk caching purposes, and system information tools will generally > indicate that this memory is in use, even though it can be > instantly claimed by any program that requests it. > > https://en.wikipedia.org/wiki/Page_cache > > If you tell a Java program that it is limited to a 2GB heap, then > that program will never use more than 2GB, plus a little extra for > the java runtime itself. I cannot give you an exact figure for > that little bit extra. But every bit of data on disk that Solr > accesses will end up (at least temporarily) in the operating > system's disk cache -- using that unallocated memory. > > https://wiki.apache.org/solr/SolrPerformanceProblems#RAM To be fair, the JVM can use *much more* memory than you have specified for your Java heap. It's just that the Java heap itself wont exceed those values. The JVM uses quite a bit of native memory which isn't counted in the Java heap. There is only one way I know of to control that, and it's to set a process-limit at the OS level on the amount of memory allowed. I'm not sure how sensitive to those limits the JVM actually is, so attempting to artificially constrain the JVM might end up with a native OOM crash. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlu/YgoACgkQHPApP6U8 pFjcbBAAgYegk20pYvfu3vcrAKxj3s+RSMGRPZ+nN5g0KYQFuhFgptYE+TqjLfBX geekQUNqNUHO5psMA5q/6m6b3LwpqrMxJiapv0wWQ2wPah21CgLs/P/iG+elNQ63 H0ZXbe3wX0P0onZbP4+sfDyzhujZ+5+gMooK87o8Q4z91hIVX1EZfM4lcaZ3pbnb JJ44YorWGPpXjQNEtOHfS7l/Q+8+6+XfEyfKha3JpRFcwcqgLpv23Koy4xgxgYr+ PMqfjptMBMjZ04xSdd491crm2yZowv3KH1Ss8v/L51rknGYPxCEkdKvPrUlpn+Rb 4WnQS6H//dJvQaLum/qR9Jxd+3vc13K7Mn++5Lu+jMbeEgaJU2hD4/ap/KMtFCqn eIXl6HQYPW36sVcm/MIpkRvAgx8vri17sd3/5sOYaETrp4SMxMN5W44GvgDdkbGF R9/tVBCFWb3p+o8eSKUf7QmARiN69DHGVwtQHWMIp8K9893IeHUNgVXKD7281zLB AjHPc7QTvAn4xne0X9lvQjr+YKOPxd9FFqMBejdKht9aBFQvApma9LtJT3FInrob QkSIx594KhoRltRy7E9t3XuWWGg8ujiuzKl6SEPsgXUC2Opwr4Wwu1yn9dCWkFJz RzCKbaDBaNmrK6HSEsoNvS+yQPksPxM8MuchFaCAMZpVOsobCM0= =77dD -END PGP SIGNATURE-
Re: Solr JVM Memory settings
Don't know if this directly affects what you're trying to do. But I have an 8GB server and when I run "solr status" I can see what % of the automatic memory allocation is being used. As it turned out, solr would occasionally exceed that (and crashed). I then began starting solr with the additional parameter: "-m 1g" Now the solr consumption is almost always 50% or less, and have had no further problems. On 10/11/2018 12:08 AM, Sourav Moitra wrote: > Hello, > > We have a Solr server with 8gb of memory. We are using solr in cloud > mode, solr version is 7.5, Java version is Oracle Java 9 and settings > for Xmx and Xms value is 2g but we are observing that the RAM getting > used to 98% when doing indexing. > > How can I ensure that SolrCloud doesn't use more than N GB of memory ? > > Sourav Moitra > https://souravmoitra.com >
Re: Solr JVM Memory settings
Beside the heap the JVM has other memory areas, like the metaspace: https://docs.oracle.com/javase/9/tools/java.htm -> MaxMetaspaceSize search for "size" in that document and you'll find tons of further settings. I have not tried out Oracle Java 9 yet. regards, Hendrik On 11.10.2018 06:08, Sourav Moitra wrote: Hello, We have a Solr server with 8gb of memory. We are using solr in cloud mode, solr version is 7.5, Java version is Oracle Java 9 and settings for Xmx and Xms value is 2g but we are observing that the RAM getting used to 98% when doing indexing. How can I ensure that SolrCloud doesn't use more than N GB of memory ? Sourav Moitra https://souravmoitra.com
Re: Solr JVM Memory settings
On 10/10/2018 11:33 PM, Sourav Moitra wrote: Hello Shawn, Thanks for quick reply. Where precisely are you seeing the 98% usage? On the Solr web UI. Although the same Solr UI is reporting heap usage to be below 1 GB. Also I found that Solr Java is holding(VIRT) 8GB of total memory even with Xmx set to 2g using top command. On the "top" screen, you can see how much memory the process is using by subtracting the value in the SHR column from the value in the RES column. The VIRT column does not indicate actual memory usage. It tells you how much *VIRTUAL* memory the process has allocated within the processor's memory mapping system. Usually, most of this value is *not* consuming actual RAM. Here's a screenshot of a "top" running on a Solr system with over 700GB of index data: https://www.dropbox.com/s/85h1weuopa643za/bigindy5-top-small-shr.png?dl=0 The VIRT column says over three quarters of a terabyte ... but the system has only 64GB of memory, 28GB of which is allocated to the Solr heap. I can guarantee you that when this screenshot was taken, the system was NOT having memory problems. If the VIRT column indicated actual memory consumption, that would be a completely impossible situation -- the machine doesn't HAVE that much memory! Additional reading on the topic of virtual memory as it relates to Lucene/Solr: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html On your system, I can pretty much guarantee that Solr is **NOT** consuming 8GB of memory. It is consuming the 2GB you told it that it could have, plus a little bit extra. If you don't have any other sizable programs running on this system, then the rest of your memory is being used by the OS -- and if you were to start another program that wanted a large chunk of memory, the OS would instantly and willingly give up some or all of its disk cache and allow that program to have the memory. Thanks, Shawn
Re: Solr JVM Memory settings
Hello Shawn, Thanks for quick reply. > > Where precisely are you seeing the 98% usage? On the Solr web UI. Although the same Solr UI is reporting heap usage to be below 1 GB. Also I found that Solr Java is holding(VIRT) 8GB of total memory even with Xmx set to 2g using top command. Can it be due to GC not clearing up the memory ? Sourav Moitra https://souravmoitra.com On Thu, Oct 11, 2018 at 10:24 AM Shawn Heisey wrote: > > On 10/10/2018 10:08 PM, Sourav Moitra wrote: > > We have a Solr server with 8gb of memory. We are using solr in cloud > > mode, solr version is 7.5, Java version is Oracle Java 9 and settings > > for Xmx and Xms value is 2g but we are observing that the RAM getting > > used to 98% when doing indexing. > > > > How can I ensure that SolrCloud doesn't use more than N GB of memory ? > > Where precisely are you seeing the 98% usage? It is completely normal > for a modern operating system to report that almost all the system > memory is in use, at least after the system has been shuffling a lot of > data. All modern operating systems will use memory that has not been > specifically allocated to programs for disk caching purposes, and system > information tools will generally indicate that this memory is in use, > even though it can be instantly claimed by any program that requests it. > > https://en.wikipedia.org/wiki/Page_cache > > If you tell a Java program that it is limited to a 2GB heap, then that > program will never use more than 2GB, plus a little extra for the java > runtime itself. I cannot give you an exact figure for that little bit > extra. But every bit of data on disk that Solr accesses will end up (at > least temporarily) in the operating system's disk cache -- using that > unallocated memory. > > https://wiki.apache.org/solr/SolrPerformanceProblems#RAM > > Thanks, > Shawn >
Re: Solr JVM Memory settings
On 10/10/2018 10:08 PM, Sourav Moitra wrote: We have a Solr server with 8gb of memory. We are using solr in cloud mode, solr version is 7.5, Java version is Oracle Java 9 and settings for Xmx and Xms value is 2g but we are observing that the RAM getting used to 98% when doing indexing. How can I ensure that SolrCloud doesn't use more than N GB of memory ? Where precisely are you seeing the 98% usage? It is completely normal for a modern operating system to report that almost all the system memory is in use, at least after the system has been shuffling a lot of data. All modern operating systems will use memory that has not been specifically allocated to programs for disk caching purposes, and system information tools will generally indicate that this memory is in use, even though it can be instantly claimed by any program that requests it. https://en.wikipedia.org/wiki/Page_cache If you tell a Java program that it is limited to a 2GB heap, then that program will never use more than 2GB, plus a little extra for the java runtime itself. I cannot give you an exact figure for that little bit extra. But every bit of data on disk that Solr accesses will end up (at least temporarily) in the operating system's disk cache -- using that unallocated memory. https://wiki.apache.org/solr/SolrPerformanceProblems#RAM Thanks, Shawn