I have 45 solr 4.1 indexes. Sizes vary between 20Gb and 2.2Gb. - 1 is 20Gb (80 million docs) - 1 is 5.1Gb (24 million docs) - 1 is 5.6Gb (26 million docs) - 1 is 6.5Gb (28 million docs) - 11 others are about 2.2Gb (6-7 million docs). - 20 others are about 600Mb (2.5 million docs)
That reminds me of something. The 4.1 indexes are 2 times smaller than the 3.5 indexes. For example the one which is 20Gb with solr 4.1 is 43Gb with solr 3.5. Maybe there is something there? There is roughly 200 queries per second. On 04/11/2013 11:07 AM, Furkan KAMACI wrote: > Hi Marc; > > Could I learn your index size and what is your performance measure as query > per second? > > 2013/4/11 Marc Des Garets <marc.desgar...@192.com> > >> Big heap because very large number of requests with more than 60 indexes >> and hundreds of million of documents (all indexes together). My problem >> is with solr 4.1. All is perfect with 3.5. I have 0.05 sec GCs every 1 >> or 2mn and 20Gb of the heap is used. >> >> With the 4.1 indexes it uses 30Gb-33Gb, the survivor space is all weird >> (it changed the size capacity to 6Mb at some point) and I have 2 sec GCs >> every minute. >> >> There must be something that has changed in 4.1 compared to 3.5 to cause >> this behavior. It's the same requests, same schemas (excepted 4 fields >> changed from sint to tint) and same config. >> >> On 04/10/2013 07:38 PM, Shawn Heisey wrote: >>> On 4/10/2013 9:48 AM, Marc Des Garets wrote: >>>> The JVM behavior is now radically different and doesn't seem to make >>>> sense. I was using ConcMarkSweepGC. I am now trying the G1 collector. >>>> >>>> The perm gen went from 410Mb to 600Mb. >>>> >>>> The eden space usage is a lot bigger and the survivor space usage is >>>> 100% all the time. >>>> >>>> I don't really understand what is happening. GC behavior really doesn't >>>> seem right. >>>> >>>> My jvm settings: >>>> -d64 -server -Xms40g -Xmx40g -XX:+UseG1GC -XX:NewRatio=1 >>>> -XX:SurvivorRatio=3 -XX:PermSize=728m -XX:MaxPermSize=728m >>> As Otis has already asked, why do you have a 40GB heap? The only way I >>> can imagine that you would actually NEED a heap that big is if your >>> index size is measured in hundreds of gigabytes. If you really do need >>> a heap that big, you will probably need to go with a JVM like Zing. I >>> don't know how much Zing costs, but they claim to be able to make any >>> heap size perform well under any load. It is Linux-only. >>> >>> I was running into extreme problems with GC pauses with my own setup, >>> and that was only with an 8GB heap. I was using the CMS collector and >>> NewRatio=1. Switching to G1 didn't help at all - it might have even >>> made the problem worse. I never did try the Zing JVM. >>> >>> After a lot of experimentation (which I will admit was not done very >>> methodically) I found JVM options that have reduced the GC pause problem >>> greatly. Below is what I am using now on Solr 4.2.1 with a total >>> per-server index size of about 45GB. This works properly on CentOS 6 >>> with Oracle Java 7u17, UseLargePages may require special kernel tuning >>> on other operating systems: >>> >>> -Xmx6144M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 >>> -XX:NewRatio=3 -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled >>> -XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts >>> >>> These options could probably use further tuning, but I haven't had time >>> for the kind of testing that will be required. >>> >>> If you decide to pay someone to make the problem going away instead: >>> >>> http://www.azulsystems.com/products/zing/whatisit >>> >>> Thanks, >>> Shawn >>> >>> >>> >> >> This transmission is strictly confidential, possibly legally privileged, >> and intended solely for the addressee. >> Any views or opinions expressed within it are those of the author and do >> not necessarily represent those of >> 192.com Ltd or any of its subsidiary companies. If you are not the >> intended recipient then you must >> not disclose, copy or take any action in reliance of this transmission. If >> you have received this >> transmission in error, please notify the sender as soon as possible. No >> employee or agent is authorised >> to conclude any binding agreement on behalf 192.com Ltd with another >> party by email without express written >> confirmation by an authorised employee of the company. >> http://www.192.com(Tel: 08000 192 192). >> 192.com Ltd is incorporated in England and Wales, company number >> 07180348, VAT No. GB 103226273. >> This transmission is strictly confidential, possibly legally privileged, and intended solely for the addressee. Any views or opinions expressed within it are those of the author and do not necessarily represent those of 192.com Ltd or any of its subsidiary companies. If you are not the intended recipient then you must not disclose, copy or take any action in reliance of this transmission. If you have received this transmission in error, please notify the sender as soon as possible. No employee or agent is authorised to conclude any binding agreement on behalf 192.com Ltd with another party by email without express written confirmation by an authorised employee of the company. http://www.192.com (Tel: 08000 192 192). 192.com Ltd is incorporated in England and Wales, company number 07180348, VAT No. GB 103226273.
