Can you verify all of your zkHost connection params across the entire cluster, and share the replicationFactor, autoAddReplicas, etc for the collection?
My theory is that you have two zookeeper configs conflicting as master elections happens, causing new replicas to get created on-the-fly. Also -- do these cores get deleted from the filesystem or left around? On Mon, May 17, 2021 at 4:11 PM Oakley, Craig (NIH/NLM/NCBI) [C] <[email protected]> wrote: > > > What does the core renames itself to, that would probably be the biggest > > hint. > > At 4:01pm 1/14/21, Solr decided on its own to drop the core > ipg_report_large_shard1_replica_n1 and to create the core > ipg_report_large_shard1_replica_n7 in its place > > At 4:33am 1/16/21, Solr decided on its own to drop the core > ipg_report_large_shard1_replica_n5 (on another node of the same SolrCloud) > and to create the core ipg_report_large_shard1_replica_n9 in its place > > At about 4:10pm 1/26/21, Solr decided on its own to drop this core > ipg_report_large_shard1_replica_n9 and to create the core > ipg_report_large_shard1_replica_n13 in its place > > In March, we created a new SolrCloud for the same collection, and reloaded > the data > > At 7:59am 5/12/21, Solr decided on its own to drop the core > ipg_report_large_shard1_replica_n1 and to create the core > ipg_report_large_shard1_replica_n5 in its place > > I am attaching an excerpt from solr.log for the most recent problem (with IP > addresses and port numbers redacted) > > Please not that Master/Slave replication breaks when a core is renamed, so > this can be a major problem > > > Any ideas? > > -----Original Message----- > From: Alexandre Rafalovitch <[email protected]> > Sent: Wednesday, May 12, 2021 2:10 PM > To: [email protected] > Subject: Re: Cores renamed > > This is truly a shot in the dark, but is it possible you have > something in core.properties file (which is where the core name is for > non-Cloud setup)? > > What does the core renames itself to, that would probably be the biggest hint. > > Regards, > Alex. > > On Wed, 12 May 2021 at 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] > <[email protected]> wrote: > > > > This phenomenon has happened again (this time without any REQUESTRECOVERY) > > > > Does anyone yet have any explanation of this? > > > > -----Original Message----- > > From: Oakley, Craig (NIH/NLM/NCBI) [C] <[email protected]> > > Sent: Thursday, January 28, 2021 10:57 AM > > To: [email protected] > > Subject: Cores renamed > > > > We recently have had a few occasions when cores for one specific collection > > were renamed (or more likely dropped and recreated, and thus ended up with > > a different core name). > > > > Is this a known phenomenon? Is there any explanation? > > > > It may be relevant that we just recently started running this SolrCloud on > > version 8.5.2, although the collection was created under Solr7.4. Also, > > this collection seems to experience some heavy updates such that the > > non-Leader replica has trouble keeping up. One of these renames occurred at > > 4:33am, so I highly suspect that the rename (or drop and recreate) was done > > by some internal Solr thread rather than by any of my coworkers. One other > > potential clue is that I can see that > > /solr/admin/cores?action=REQUESTRECOVERY was usually run on the new core a > > moment after it was created. > > > > Does anyone have any insights?
