Hello Erick, I know the article, it is about virtual memory. My problem is with shared memory. Correct me if i am wrong, but MMApped files do not occupy shared but virtual instead. If i am wrong, the article must be rewritten. Our main searchers show very normal numbers for virtual, which is index size plus RSS.
My problem is that RSS is more than four times our heap size. Usually RSS is heap + PermGen/metaspace + code cache + (threads * stack size) + some other stuff, so a 1 GB heap would consume about 1500 MB tops. Our RSS for Solr is 4.3 GB of which 2.85 GB is shared memory. Let me know what you think. Thanks, Markus -----Original message----- > From:Erick Erickson <erickerick...@gmail.com> > Sent: Thursday 24th August 2017 3:35 > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: Solr uses lots of shared memory! > > I suspect you've already seen this, but top and similar can be > confusing when trying to interpret MMapDirectory. Uwe has an excellent > explication: > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Best, > Erick > > On Wed, Aug 23, 2017 at 9:10 AM, Markus Jelsma > <markus.jel...@openindex.io> wrote: > > I have the problem in production and local, with default Solr 6.6 JVM > > arguments, environments are: > > > > openjdk version "1.8.0_141" > > OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15) > > OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode) > > Linux idx1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1 (2017-06-18) x86_64 > > GNU/Linux > > > > and > > > > openjdk version "1.8.0_131" > > OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.17.04.3-b11) > > OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) > > Linux midas 4.10.0-32-generic #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 > > x86_64 x86_64 x86_64 GNU/Linux > > > > Regarding the node that shows the problem, can you reproduce it locally? > > Fire it up, put some data in, confirm low shared space usage, and execute > > few thousands queries against it? We immediately see a sharp rise in shared > > memory, MB's per second until it reaches some sort of plateau. > > > > -----Original message----- > >> From:Shawn Heisey <apa...@elyograg.org> > >> Sent: Wednesday 23rd August 2017 16:37 > >> To: solr-user@lucene.apache.org > >> Subject: Re: Solr uses lots of shared memory! > >> > >> On 8/23/2017 7:32 AM, Markus Jelsma wrote: > >> > Why does it slowly increase over time? > >> > Why does it appear to correlate to index size? > >> > Is anyone else seeing this on their 6.6 cloud production or local > >> > machines? > >> > >> More detailed information included here. My 6.6 dev install is NOT > >> having the problem, but a much older version IS. > >> > >> I grabbed this screenshot only moments ago from a production server > >> which is exhibiting a large SHR value for the Solr process: > >> > >> https://www.dropbox.com/s/q79lo2gft9es06u/idxa1-top-big-shr.png?dl=0 > >> > >> This is Solr 4.7.2, with a 10 month uptime for the Solr process, running > >> with these arguments: > >> > >> -DSTOP.KEY=REDACTED > >> -DSTOP.PORT=8078 > >> -Djetty.port=8981 > >> -Dsolr.solr.home=/index/solr4 > >> -Dcom.sun.management.jmxremote.authenticate=false > >> -Dcom.sun.management.jmxremote.ssl=false > >> -Dcom.sun.management.jmxremote.port=8686 > >> -Dcom.sun.management.jmxremote > >> -XX:+PrintReferenceGC > >> -XX:+PrintAdaptiveSizePolicy > >> -XX:+PrintGCDetails > >> -XX:+PrintGCDateStamps > >> -Xloggc:logs/gc.log-verbose:gc > >> -XX:+AggressiveOpts > >> -XX:+UseLargePages > >> -XX:InitiatingHeapOccupancyPercent=75 > >> -XX:MaxGCPauseMillis=250 > >> -XX:G1HeapRegionSize=8m > >> -XX:+ParallelRefProcEnabled > >> -XX:+PerfDisableSharedMem > >> -XX:+UseG1GC > >> -Dlog4j.configuration=file:etc/log4j.properties > >> -Xmx8192M > >> -Xms4096M > >> > >> The OS is CentOS 6, with the following Java and kernel: > >> > >> java version "1.7.0_72" > >> Java(TM) SE Runtime Environment (build 1.7.0_72-b14) > >> Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) > >> > >> Linux idxa1 2.6.32-431.11.2.el6.centos.plus.x86_64 #1 SMP Tue Mar 25 > >> 21:36:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > >> I also just grabbed a screenshot from my dev server, running Ubuntu 14, > >> Solr 6.6.0, a LOT more index data, and a more recent Java version. Solr > >> has an uptime of about one month. This server was installed with the > >> service installer script, so it uses bin/solr. It doesn't seem to have > >> the same problem: > >> > >> https://www.dropbox.com/s/85h1weuopa643za/bigindy5-top-small-shr.png?dl=0 > >> > >> java version "1.8.0_144" > >> Java(TM) SE Runtime Environment (build 1.8.0_144-b01) > >> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) > >> > >> Linux bigindy5 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC > >> 2016 x86_64 x86_64 x86_64 GNU/Linux > >> > >> The arguments for this one are very similar to the production server: > >> > >> -DSTOP.KEY=solrrocks > >> -DSTOP.PORT=7982 > >> -Dcom.sun.management.jmxremote > >> -Dcom.sun.management.jmxremote.authenticate=false > >> -Dcom.sun.management.jmxremote.local.only=false > >> -Dcom.sun.management.jmxremote.port=18982 > >> -Dcom.sun.management.jmxremote.rmi.port=18982 > >> -Dcom.sun.management.jmxremote.ssl=false > >> -Djetty.home=/opt/solr6/server > >> -Djetty.port=8982 > >> -Dlog4j.configuration=file:/index/solr6/log4j.properties > >> -Dsolr.install.dir=/opt/solr6 > >> -Dsolr.log.dir=/index/solr6/logs > >> -Dsolr.log.muteconsole > >> -Dsolr.solr.home=/index/solr6/data > >> -Duser.timezone=UTC > >> -XX:+AggressiveOpts > >> -XX:+ParallelRefProcEnabled > >> -XX:+PrintGCApplicationStoppedTime > >> -XX:+PrintGCDateStamps > >> -XX:+PrintGCDetails > >> -XX:+PrintGCTimeStamps > >> -XX:+PrintHeapAtGC > >> -XX:+PrintTenuringDistribution > >> -XX:+UseG1GC > >> -XX:+UseGCLogFileRotation > >> -XX:+UseLargePages > >> -XX:G1HeapRegionSize=8m > >> -XX:GCLogFileSize=20M > >> -XX:InitiatingHeapOccupancyPercent=75 > >> -XX:MaxGCPauseMillis=250 > >> -XX:NumberOfGCLogFiles=9 > >> -XX:OnOutOfMemoryError=/opt/solr6/bin/oom_solr.sh 8982 /index/solr6/logs > >> -Xloggc:/index/solr6/logs/solr_gc.log > >> -Xms28g > >> -Xmx28g > >> -Xss256k > >> -verbose:gc > >> > >> Neither system has any huge pages allocated in the OS, so I doubt that > >> the UseLargePages option is actually doing anything. I've left it there > >> in case I *do* enable huge pages, so they will automatically get used. > >> > >> Thanks, > >> Shawn > >> > >> >
