It could be interesting to explore optimizing "openNewSearcher" when there is an existing searcher open over some of the same segments. For such older segments, after replication, are they at the same exact file path or does replication create a new path? I forget this detail; I recall seeing some index data path shuffling going on. ReplicationHandler etc. is overly complex.
~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Wed, Nov 23, 2022 at 1:50 PM Patson Luk <pat...@fullstory.com> wrote: > Hi all! > > We are testing multiple replica setup here (1 NRT + 1 PULL) and noticed > that CPU consumption for replication is unreasonably high. Profiling shows > that `SolrCore#openNewSearcher` triggered from `IndexFetcher` takes much > more CPU time than the same method triggered from regular commits. > > Debugging shows that when `SolrCore#openNewSearcher` is triggered from > `IndexFetcher`, it opens a new `SegmentReader` for every single fragment > for the updated collection. As a new `IndexWriter`, which keeps a > `ReaderPool`, is instantiated for each replication. And such pool is not > reused nor previous segment readers are carried over. > > Details in this ticket https://issues.apache.org/jira/browse/SOLR-16560. > > Since I'm pretty new to this area, I would love to get some thoughts from > the community! > > Many thanks! > Patsn >
