Hi Shawn , Thank you for the pointers , here is the information :
What OS is Solr running on? I'm only asking because some additional information I'm after has different gathering methods depending on OS. Other questions: *OpenJDK 64-Bit Server VM (25.141-b16) for linux-amd64 JRE (1.8.0_141-b16), built on Jul 20 2017 21:47:59 by "mockbuild" with gcc 4.4.7 20120313 (Red Hat 4.4.7-18)* *Memory: 4k page, physical 264477520k(92198808k free), swap 0k(0k free)* Is there only one Solr process per machine, or more than one? *On an average yes , one solr process per machine , however , we do have a machine (where this log is taken) has two solr processes (master and slave)* How many total documents are managed by one machine? *About 220945 per machine ( and double for this machine as it has instance of master as well as other slave)* How big is all the index data managed by one machine? *The index is about 4G.* What is the max heap on each Solr process? *Max heap is 25G for each Solr Process. (Xms 25g Xmx 25g)* The reason of choosing RAMDirectory was that it was used in the similar manner while the production Solr was on Version 4.3.2, so no particular reason but just replicated how it was working , never thought this may give troubles. I had included a pastebin of GC snapshot (the complete log was too big to be included in the pastebin , so pasted a sampler) Another thing is as we observed the CPU cycles yesterday in high load condition we observed that the Highlighter component was taking longest , is there anything in particular we forgot to include that highlighting doesn't gives a performance hit . Attached is the snapshot taken from jvisualvm. Please guide. Thanks, Atita On Thu, Jul 27, 2017 at 2:46 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 7/26/2017 1:49 AM, Atita Arora wrote: > > We did our functional and load testing on these boxes , however when we > > released it to production along with the same application (using SolrJ to > > query Solr) , we ran into severe CPU issues. > > Just to add we're on Master - Slave where master has index on > > NRTCachingDirectory > > and Slave on RAMDirectory. > > > > As soon as we placed the slaves under load balancer , under NO LOAD > > condition as well , the slave went from a load of 4 -> 10 -> 16 - > 100 > in > > 12 mins. > > > > I suspected this to be caused due to replication but this is never > ending , > > so before this crashed we de-provisioned it and brought it down. > > > > I'm not sure what could possibly cause it. > > > > I looked into the caches , where documentcache , filtercache , > > queryresultcaches are set to defaults 1024 and 100 documents. > > > > I tried observing the GC activity on GCViewer too , which does'nt really > > shows something alarming (as in what I feel) - a sampler at > > https://pastebin.com/cnuATYrS > > What OS is Solr running on? I'm only asking because some additional > information I'm after has different gathering methods depending on OS. > Other questions: > > Is there only one Solr process per machine, or more than one? > How many total documents are managed by one machine? > How big is all the index data managed by one machine? > What is the max heap on each Solr process? > > FYI, RAMDirectory is not the preferred way of running Solr or Lucene. > If you have enough memory to hold the entire index, it's better to let > the OS handle keeping that information in memory, rather than having > Lucene and Java do it. > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > NRTCachingDirectoryFactory uses MMap by default as its delegate > implementation, so your master is fine. > > I would be interested in getting a copy of Solr's gc log from a system > with high CPU to look at. > > Thanks, > Shawn > >
