[
https://issues.apache.org/jira/browse/SOLR-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686906#comment-16686906
]
bidorbuy commented on SOLR-11078:
---------------------------------
Hi all,
ex CTO from bidorbuy here. I had planned to migrate to 7.5 before I am leaving
the company end of November, but other priorities took over.
Curiosity still got the better of me and over the last week I pulled up three
7.5. instances running on 8 cores, 10GB RAM (the Solr- and JVM tuning is
unchanged from the 2017 configs I attached in the original post).
I apologise for the lack of depth of the analysis - it is not how I typically
work, but instead of not providing a follow-up I thought "better little than
nothing". I am sure that my successor / team will follow up when they have an
in-depth analysis by January 2019.
The test was done by replaying Solr queries from an existing 7.4. Solr install
across the 7.5 nodes and replayed the same production-load on different schemas
using the main-index which consists of 7,5m documents and a size of 4GB:
* Green: Now deprecated Trie*-fields (this is what we currently running in
production)
* Orange: Point*-fields
* Blue: StrField
Trie*-fields (Green or bottom line) still produce overall lowest CPU and memory
utilisation. Throughput: 19,5m queries, 21ms avg query time, 7.3sec max query
time
Point*-fields (Orange or middle line) consumed more resources. Throughput:
18,6m queries, 35ms avg query time, 10sec max query time
StrField (Blue or top-line) was visibly using more resources. GC was also more
frequent. Throughput: 17,6m queries, 42ms avg query time, 13sec max query time
Admittedly, I did not tune JVM (using G1) and I am sure that if I had the time
I could have tuned resource utilisation better. I doubt that such tuning would
have dramatically improved throughput or query time.
Perhaps I misunderstood this, but I thought Point-field types would outperform
Trie*fields in range queries. In my tests I did not notice a dramatic
difference for range queries when using Point*fields. I did however notice that
some simple field/value queries underperformed when using Point*-fields over
Trie*fields.
I know it was mentioned that if Point-fields are not performing we should use
StrFields, and maybe I missed a fundamental configuration step, but StrField
generally performed worse than Point or Trie fields.
I wish I had more time to analyse queries in more detail and perhaps play with
variations in the schema to possibly assist with an improvement in this. As you
can see from our schema, it is pretty much "vanilla" and I am surprised that
other users have not raised issues - in our business (ecommerce) a difference
of 14ms in average queries (or a variance of a good 2.7 seconds in poor
performing queries) has a dramatic result on performance. Perhaps in other use
cases such small variances between field-types do not matter.
!image-2018-11-14-19-02-20-216.png!
I am unfortunately not going to be able to contribute to this issue any more as
I will not have access to the Solr environments.
I would like to thank all Solr contributors for your outstanding work and deep
level of knowledge and commitment as well as the helping hand you lend in
assisting us with questions, issues and patches. I am grateful to have had the
opportunity to use Solr when it started "becoming mainstream" as version 3.6
and thank you all for your contributions! All the best ~ Gerd Naschenweng.
> Solr query performance degradation since Solr 6.4.2
> ---------------------------------------------------
>
> Key: SOLR-11078
> URL: https://issues.apache.org/jira/browse/SOLR-11078
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search, Server
> Affects Versions: 6.6, 7.1
> Environment: * CentOS 7.3 (Linux zasolrm03 3.10.0-514.26.2.el7.x86_64
> #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)
> * Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
> * 4 CPU, 10GB RAM
> Running Solr 6.6.0 with the following JVM settings:
> java -server -Xms4G -Xmx4G -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/home/prodza/solrserver/../logs/solr_gc.log -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
> -Dsolr.log.dir=/home/prodza/solrserver/../logs -Djetty.port=8983
> -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=SAST
> -Djetty.home=/home/prodza/solrserver/server
> -Dsolr.solr.home=/home/prodza/solrserver/../solr
> -Dsolr.install.dir=/home/prodza/solrserver
> -Dlog4j.configuration=file:/home/prodza/solrserver/../config/log4j.properties
> -Xss256k -Xss256k -Dsolr.log.muteconsole
> -XX:OnOutOfMemoryError=/home/prodza/solrserver/bin/oom_solr.sh 8983
> /home/prodza/solrserver/../logs -jar start.jar --module=http
> Reporter: bidorbuy
> Priority: Major
> Attachments: compare-6.4.2-6.6.0.png, core-admin-tradesearch.png,
> image-2018-11-14-19-00-39-395.png, image-2018-11-14-19-02-20-216.png,
> jvm-stats.png, schema.xml, screenshot-1.png, screenshot-2.png,
> screenshot-3.png, solr-6-4-2-schema.xml, solr-6-4-2-solrconfig.xml,
> solr-7-1-0-managed-schema, solr-7-1-0-solrconfig.xml, solr-71-vs-64.png,
> solr-sample-warning-log.txt, solr.in.sh, solrconfig.xml
>
>
> We are currently running 2 separate Solr servers - refer to screenshots:
> * zasolrm02 is running on Solr 6.4.2
> * zasolrm03 is running on Solr 6.6.0
> Both servers have the same OS / JVM configuration and are using their own
> indexes. We round-robin load-balance through our Tomcats and notice that
> Since Solr 6.4.2 performance has dropped. We have two indices per server
> "searchsuggestions" and "tradesearch". There is a noticeable drop in
> performance since Solr 6.4.2.
> I am not sure if this is perhaps related to metric collation or other
> underlying changes. I am not sure if other high transaction users have
> noticed similar issues.
> *1) zasolrm03 (6.6.0) is almost twice as slow on the tradesearch index:*
> !compare-6.4.2-6.6.0.png!
> *2) This is also visible in the searchsuggestion index:*
> !screenshot-1.png!
> *3) The Tradesearch index shows the biggest difference:*
> !screenshot-2.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
