[ https://issues.apache.org/jira/browse/SOLR-10130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870751#comment-15870751 ]
Walter Underwood commented on SOLR-10130: ----------------------------------------- This might be part of it: [wunder@new-solr-c01.test3]# ls -lh /solr/data/questions_shard2_replica1/data/tlog/ total 4.7G -rw-r--r-- 1 bin bin 4.7G Feb 13 11:04 tlog.0000000000000000000 [wunder@new-solr-c01.test3]# du -sh /solr/data/questions_shard2_replica1/data/* 8.4G /solr/data/questions_shard2_replica1/data/index 4.0K /solr/data/questions_shard2_replica1/data/snapshot_metadata 4.7G /solr/data/questions_shard2_replica1/data/tlog Last Modified: 3 days ago Num Docs: 3683075 Max Doc: 3683075 Heap Memory Usage: -1 Deleted Docs: 0 Version: 2737 Segment Count: 26 Optimized: yes Current: yes > Serious performance degradation in Solr 6.4.1 due to the new metrics > collection > ------------------------------------------------------------------------------- > > Key: SOLR-10130 > URL: https://issues.apache.org/jira/browse/SOLR-10130 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics > Affects Versions: 6.4.1 > Environment: Centos 7, OpenJDK 1.8.0 update 111 > Reporter: Ere Maijala > Assignee: Andrzej Bialecki > Priority: Blocker > Labels: perfomance > Fix For: master (7.0), 6.4.2 > > Attachments: SOLR-10130.patch, SOLR-10130.patch, > solr-8983-console-f1.log > > > We've stumbled on serious performance issues after upgrading to Solr 6.4.1. > Looks like the new metrics collection system in MetricsDirectoryFactory is > causing a major slowdown. This happens with an index configuration that, as > far as I can see, has no metrics specific configuration and uses > luceneMatchVersion 5.5.0. In practice a moderate load will completely bog > down the server with Solr threads constantly using up all CPU (600% on 6 core > machine) capacity with a load that normally where we normally see an average > load of < 50%. > I took stack traces (I'll attach them) and noticed that the threads are > spending time in com.codahale.metrics.Meter.mark. I tested building Solr > 6.4.1 with the metrics collection disabled in MetricsDirectoryFactory getByte > and getBytes methods and was unable to reproduce the issue. > As far as I can see there are several issues: > 1. Collecting metrics on every single byte read is slow. > 2. Having it enabled by default is not a good idea. > 3. The comment "enable coarse-grained metrics by default" at > https://github.com/apache/lucene-solr/blob/branch_6x/solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java#L104 > implies that only coarse-grained metrics should be enabled by default, and > this contradicts with collecting metrics on every single byte read. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org