[ https://issues.apache.org/jira/browse/SOLR-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240871#comment-16240871 ]
David Smiley commented on SOLR-11078: ------------------------------------- bq. Is there more documentation available about *Point-types? I'm not sure. I'll update the ref guide after you take the changes for a whirl to confirm what I expect will happen. bq. I am unsure what it means from a performance perspective that Point-types do not support FieldCache and as such we need to use doc-values for faceting to be supported. Curious; can you provide a reference to where you've read that about the FieldCache? It's right but just wanted to see where that advise is; I didn't notice it in the ref guide. Any way... I suspect you've _already_ (in 6.x) been using docValues=true on appropriate fields since you didn't mention to the contrary. So there's no change for you in this respect. Basically, know that Solr needs fast access to values on a doc/field basis for certain non-search features. Generally, the ideal way to make that available is to set docValues=true on the field. Alternatively if you don't, and if the field type supports "uninversion" (Trie fields & StrField do, Point fields don't) then Solr will use it's so-called FieldCache which un-inverts the index=true data into memory. Lets say you weren't using docValues at all and now you do. The net perceived change is * smaller java heap requirements; this data is accessed off-heap in memory mapped IO instead. So you still need same RAM roughly but Java GC can have an easier time. * better realtime search. The warmup queries (generally user-configured to run after commit in solrconfig) should be faster to complete if they use faceting, sorting etc. since they won't need to trigger the uninversion of index structures into memory. If you don't have warmup, then the first user's search query to come along after a commit used to be hit with the uninversion penalty and now it won't (so faster queries at this 99 percentile). * larger disk requirements to hold the docValues data General access performance isn't necessarily better though; in some cases it has been shown to be worse :-/ bq. One idea we had is to change all non-range fields to StrField (one by one) and measure performance impact and then start switching *Point-fields. I am not sure if there is a better/scientific way to go about this (analysing the query performance is difficult as queries are typically some text-search with filtering/faceting - so not many unique queries where it is easy to show some trends). Benchmarking/analysis can be a real time sink; I suppose what you should do depends on what you hope to get out of it. Assuming you hope to not use deprecated Trie fields... then yes moving away from them is one goal. And getting equal or better performance is another goal. I doubt you need to get any more precise than that unless you are digging deeper to discover why one of those goals isn't met (gets slower). As you've indicated switching the "non-range" usages to StrField is one step. My expectation of the result is that it'll be a non-event. StrField loses the typed ergonomics of Trie but otherwise should be very similar for non-range use. I wouldn't bother changing one field at a time and measuring in-between... that seems like a lot of time; although if you discover some unexpected bad results then you could do this to help deduce which field & query exhibits the problem. RE the range fields, yep, them from TrieFields to PointFields (all at once) is the next step. This one may be interesting; hopefully nothing gets worse. If it's better than declare success and move on! > Solr query performance degradation since Solr 6.4.2 > --------------------------------------------------- > > Key: SOLR-11078 > URL: https://issues.apache.org/jira/browse/SOLR-11078 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: search, Server > Affects Versions: 6.6, 7.1 > Environment: * CentOS 7.3 (Linux zasolrm03 3.10.0-514.26.2.el7.x86_64 > #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux) > * Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) > * 4 CPU, 10GB RAM > Running Solr 6.6.0 with the following JVM settings: > java -server -Xms4G -Xmx4G -XX:NewRatio=3 -XX:SurvivorRatio=4 > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC > -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 > -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m > -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 > -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled > -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime > -Xloggc:/home/prodza/solrserver/../logs/solr_gc.log -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M > -Dsolr.log.dir=/home/prodza/solrserver/../logs -Djetty.port=8983 > -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=SAST > -Djetty.home=/home/prodza/solrserver/server > -Dsolr.solr.home=/home/prodza/solrserver/../solr > -Dsolr.install.dir=/home/prodza/solrserver > -Dlog4j.configuration=file:/home/prodza/solrserver/../config/log4j.properties > -Xss256k -Xss256k -Dsolr.log.muteconsole > -XX:OnOutOfMemoryError=/home/prodza/solrserver/bin/oom_solr.sh 8983 > /home/prodza/solrserver/../logs -jar start.jar --module=http > Reporter: bidorbuy > Attachments: compare-6.4.2-6.6.0.png, core-admin-tradesearch.png, > jvm-stats.png, schema.xml, screenshot-1.png, screenshot-2.png, > solr-6-4-2-schema.xml, solr-6-4-2-solrconfig.xml, solr-7-1-0-managed-schema, > solr-7-1-0-solrconfig.xml, solr-71-vs-64.png, solr-sample-warning-log.txt, > solr.in.sh, solrconfig.xml > > > We are currently running 2 separate Solr servers - refer to screenshots: > * zasolrm02 is running on Solr 6.4.2 > * zasolrm03 is running on Solr 6.6.0 > Both servers have the same OS / JVM configuration and are using their own > indexes. We round-robin load-balance through our Tomcats and notice that > Since Solr 6.4.2 performance has dropped. We have two indices per server > "searchsuggestions" and "tradesearch". There is a noticeable drop in > performance since Solr 6.4.2. > I am not sure if this is perhaps related to metric collation or other > underlying changes. I am not sure if other high transaction users have > noticed similar issues. > *1) zasolrm03 (6.6.0) is almost twice as slow on the tradesearch index:* > !compare-6.4.2-6.6.0.png! > *2) This is also visible in the searchsuggestion index:* > !screenshot-1.png! > *3) The Tradesearch index shows the biggest difference:* > !screenshot-2.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org