We have the following setup , solr 7.7.2 with 1 TLOG Leader & 1 TLOG replica with a single shard. We have about 34.5 million documents with an approximate index size of 600GB. I have noticed a degraded query performance whenever the replica is trying to (guessing here) sync or perform actual replication. To test this, I fire a very basic query using solrj client & the query comes back right away, but whenever the replication is trying to see how far behind it is by comparing the generation ids the same queries take longer. In production we do not make these simple queries, but rather complex queries with filter queries & sorting. These queries take too long as compared to our previous (standalone solr 6.1.0)
Any help here is appreciated 20-09-02 16:35:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=0 2020-09-02 16:35:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=0 2020-09-02 16:36:00 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=0 2020-09-02 16:36:00 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=0 2020-09-02 16:36:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=0 2020-09-02 16:36:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=0 *2020-09-02 16:37:01* INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=*1011* *2020-09-02 16:37:01* INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909 status=0 QTime=*758* *2020-09-02 16:37:32* INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458957 status=0 QTime=*1077* *2020-09-02 16:37:32* INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458957 status=0 QTime=*1081* 2020-09-02 16:38:02 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458957 status=0 QTime=*668* 2020-09-02 16:38:03 INFO [db_shard1_replica_t3] webapp=/solr path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458957 status=0 QTime=*1001* *2020-09-02 16:37:01* INFO Master's generation: 263116 *2020-09-02 16:37:01* INFO Master's version: 1599064577322 *2020-09-02 16:37:01* INFO Slave's generation: 263116 *2020-09-02 16:37:01* INFO Slave's version: 1599064577322 *2020-09-02 16:37:01* INFO Slave in sync with master. 2020-09-02 16:37:02 INFO Master's generation: 104189 2020-09-02 16:37:02 INFO Master's version: 1599064620532 2020-09-02 16:37:02 INFO Slave's generation: 104188 2020-09-02 16:37:02 INFO Slave's version: 1599064560341 2020-09-02 16:37:02 INFO Starting replication process 2020-09-02 16:37:02 INFO Number of files in latest index in master: 1010 2020-09-02 16:37:02 INFO Starting download (fullCopy=false) to NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/test_shard1_replica_t3/data/index.20200902163702345 lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee; maxCacheMB=48.0 maxMergeSizeMB=4.0) 2020-09-02 16:37:02 INFO Bytes downloaded: 837587, Bytes skipped downloading: 0 2020-09-02 16:37:02 INFO Total time taken for download (fullCopy=false,bytesDownloaded=837587) : 0 secs (null bytes/sec) to NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/test_shard1_replica_t3/data/index.20200902163702345 lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee; maxCacheMB=48.0 maxMergeSizeMB=4.0) 2020-09-02 16:37:03 INFO New IndexWriter is ready to be used. 2020-09-02 16:37:03 INFO Master's generation: 124002 2020-09-02 16:37:03 INFO Master's version: 1599064617242 2020-09-02 16:37:03 INFO Slave's generation: 124000 2020-09-02 16:37:03 INFO Slave's version: 1599064492914 2020-09-02 16:37:03 INFO Starting replication process 2020-09-02 16:37:04 INFO [db_shard1_replica_t3] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from= http://178.33.234.1:8983/solr/db_shard1_replica_t25/&wt=javabin&version=2}{add=[11907382419 (1676740784884285440), 11907383701 (1676740784889528320), 11907383253 (1676740784900014080), 11907379290 (1676740785002774528), 11907382623 (1676740785005920256), 11907378461 (1676740785011163136), 11907382429 (1676740785012211712), 11907380739 (1676740785023746048), 11907381184 (1676740785038426112), 11907380614 (1676740785042620416), ... (36 adds)]} 0 2227 2020-09-02 16:37:04 INFO Number of files in latest index in master: 1227 2020-09-02 16:37:04 INFO [db_shard1_replica_t3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http:// 178.33.234.1:8983/solr/db_shard1_replica_t25/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit= <http://178.33.234.1:8983/solr/db_shard1_replica_t25/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false%7D%7Bcommit=>} 0 0 2020-09-02 16:37:04 INFO Starting download (fullCopy=false) to NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/db_shard1_replica_t3/data/index.20200902163704048 lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee; maxCacheMB=48.0 maxMergeSizeMB=4.0) 2020-09-02 16:37:04 INFO [db_shard1_replica_t3] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http:// 178.33.234.1:8983/solr/db_shard1_replica_t25/&wt=javabin&version=2}{add=[10350978095 <http://178.33.234.1:8983/solr/db_shard1_replica_t25/&wt=javabin&version=2%7D%7Badd=%5B10350978095> (1676740051134840833), 10350978170 (1676740051136937985), 10350978182 (1676740051137986560), 10350978368 (1676740051140083713), 10350978377 (1676740051142180865), 10350979169 (1676740051153715200), 10350979195 (1676740051380207616), 10350979757 (1676740051398033408), 10350980088 (1676740051407470593), 10350980128 (1676740051410616321), ... (50000 adds)]} 0 702638