I suspect you've already seen this, but top and similar can be confusing when trying to interpret MMapDirectory. Uwe has an excellent explication:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Best, Erick On Wed, Aug 23, 2017 at 9:10 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: > I have the problem in production and local, with default Solr 6.6 JVM > arguments, environments are: > > openjdk version "1.8.0_141" > OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15) > OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode) > Linux idx1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1 (2017-06-18) x86_64 > GNU/Linux > > and > > openjdk version "1.8.0_131" > OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.17.04.3-b11) > OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) > Linux midas 4.10.0-32-generic #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > > Regarding the node that shows the problem, can you reproduce it locally? Fire > it up, put some data in, confirm low shared space usage, and execute few > thousands queries against it? We immediately see a sharp rise in shared > memory, MB's per second until it reaches some sort of plateau. > > -----Original message----- >> From:Shawn Heisey <apa...@elyograg.org> >> Sent: Wednesday 23rd August 2017 16:37 >> To: solr-user@lucene.apache.org >> Subject: Re: Solr uses lots of shared memory! >> >> On 8/23/2017 7:32 AM, Markus Jelsma wrote: >> > Why does it slowly increase over time? >> > Why does it appear to correlate to index size? >> > Is anyone else seeing this on their 6.6 cloud production or local machines? >> >> More detailed information included here. My 6.6 dev install is NOT >> having the problem, but a much older version IS. >> >> I grabbed this screenshot only moments ago from a production server >> which is exhibiting a large SHR value for the Solr process: >> >> https://www.dropbox.com/s/q79lo2gft9es06u/idxa1-top-big-shr.png?dl=0 >> >> This is Solr 4.7.2, with a 10 month uptime for the Solr process, running >> with these arguments: >> >> -DSTOP.KEY=REDACTED >> -DSTOP.PORT=8078 >> -Djetty.port=8981 >> -Dsolr.solr.home=/index/solr4 >> -Dcom.sun.management.jmxremote.authenticate=false >> -Dcom.sun.management.jmxremote.ssl=false >> -Dcom.sun.management.jmxremote.port=8686 >> -Dcom.sun.management.jmxremote >> -XX:+PrintReferenceGC >> -XX:+PrintAdaptiveSizePolicy >> -XX:+PrintGCDetails >> -XX:+PrintGCDateStamps >> -Xloggc:logs/gc.log-verbose:gc >> -XX:+AggressiveOpts >> -XX:+UseLargePages >> -XX:InitiatingHeapOccupancyPercent=75 >> -XX:MaxGCPauseMillis=250 >> -XX:G1HeapRegionSize=8m >> -XX:+ParallelRefProcEnabled >> -XX:+PerfDisableSharedMem >> -XX:+UseG1GC >> -Dlog4j.configuration=file:etc/log4j.properties >> -Xmx8192M >> -Xms4096M >> >> The OS is CentOS 6, with the following Java and kernel: >> >> java version "1.7.0_72" >> Java(TM) SE Runtime Environment (build 1.7.0_72-b14) >> Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) >> >> Linux idxa1 2.6.32-431.11.2.el6.centos.plus.x86_64 #1 SMP Tue Mar 25 >> 21:36:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> I also just grabbed a screenshot from my dev server, running Ubuntu 14, >> Solr 6.6.0, a LOT more index data, and a more recent Java version. Solr >> has an uptime of about one month. This server was installed with the >> service installer script, so it uses bin/solr. It doesn't seem to have >> the same problem: >> >> https://www.dropbox.com/s/85h1weuopa643za/bigindy5-top-small-shr.png?dl=0 >> >> java version "1.8.0_144" >> Java(TM) SE Runtime Environment (build 1.8.0_144-b01) >> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) >> >> Linux bigindy5 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC >> 2016 x86_64 x86_64 x86_64 GNU/Linux >> >> The arguments for this one are very similar to the production server: >> >> -DSTOP.KEY=solrrocks >> -DSTOP.PORT=7982 >> -Dcom.sun.management.jmxremote >> -Dcom.sun.management.jmxremote.authenticate=false >> -Dcom.sun.management.jmxremote.local.only=false >> -Dcom.sun.management.jmxremote.port=18982 >> -Dcom.sun.management.jmxremote.rmi.port=18982 >> -Dcom.sun.management.jmxremote.ssl=false >> -Djetty.home=/opt/solr6/server >> -Djetty.port=8982 >> -Dlog4j.configuration=file:/index/solr6/log4j.properties >> -Dsolr.install.dir=/opt/solr6 >> -Dsolr.log.dir=/index/solr6/logs >> -Dsolr.log.muteconsole >> -Dsolr.solr.home=/index/solr6/data >> -Duser.timezone=UTC >> -XX:+AggressiveOpts >> -XX:+ParallelRefProcEnabled >> -XX:+PrintGCApplicationStoppedTime >> -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails >> -XX:+PrintGCTimeStamps >> -XX:+PrintHeapAtGC >> -XX:+PrintTenuringDistribution >> -XX:+UseG1GC >> -XX:+UseGCLogFileRotation >> -XX:+UseLargePages >> -XX:G1HeapRegionSize=8m >> -XX:GCLogFileSize=20M >> -XX:InitiatingHeapOccupancyPercent=75 >> -XX:MaxGCPauseMillis=250 >> -XX:NumberOfGCLogFiles=9 >> -XX:OnOutOfMemoryError=/opt/solr6/bin/oom_solr.sh 8982 /index/solr6/logs >> -Xloggc:/index/solr6/logs/solr_gc.log >> -Xms28g >> -Xmx28g >> -Xss256k >> -verbose:gc >> >> Neither system has any huge pages allocated in the OS, so I doubt that >> the UseLargePages option is actually doing anything. I've left it there >> in case I *do* enable huge pages, so they will automatically get used. >> >> Thanks, >> Shawn >> >>
