Speaking of backing up ZK, are people doing this now for Solr and can information be shared on the approach? I'm aware of scripts/code out there that can save ZK's contents to local disk and reconstitute back to ZK, perhaps at another cluster and/or chroot. But I'm not aware of what issues there may be in using this in conjunction with Solr. Perhaps "live nodes" needs to be edited to reflect possibly new Solr live nodes.
~ David On Thu, Jun 18, 2020 at 8:39 AM Erick Erickson <[email protected]> wrote: > I’d add to Shalin’s comment that your client should MIGRATESTATEFORMAT > after > the process is done, turn off legacyCloud and insure that > /clusterstate.json is > empty after that’s done. > > The old behavior restored all the state information to /clusterstate.json, > and > you then have this weird split setup where new collections are in the > proper > /collections/my_collection/state.json and the recovered bits are in > /clusterstate.json. > > And clusterstate.json is removed in 9.0 so this won’t work in future. > > Another way to do this: > > 1. create an empty collection as Shalin outlined, _but_ leader-only. > > 2. copy the data dir from one replica from each shard to the data dir of > the new collection > (data/tlog and data/index, although I don’t really think tlog is necessary) > > 3. start up Solr and verify it’s as you expect. > > 4. build out the collection by using ADDREPLICA if you need more replicas. > > 5. make it a practice to back up the zk data ;) > > Be very, very sure you copy the data (tlog + index) to corresponding > shards, > i.e. shard1old->shard1new. > > And yes, the old versions of Solr did this, but it lead to endless > problems. If the customer was > relying on that behavior, it was a mistake. > > > On Jun 18, 2020, at 7:03 AM, Shalin Shekhar Mangar < > [email protected]> wrote: > > > > Hi Mikhail, > > > > 1. Create an empty collection with the same number of shards as before > > 2. Set legacyCloud cluster property to true (it is false by default) > using: ./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 > -cmd clusterprop -name legacyCloud -val true > > 3. Bring your solr nodes back online, they should register themselves to > ZK as they used to do in old versions of Solr > > > > But as always in such cases, take snapshots of those persistent disks > and test before you execute :) > > > > Also, be aware of this bug: > https://issues.apache.org/jira/browse/SOLR-11503 when in future you want > to turn off legacyCloud mode. > > > > Good luck! > > > > On Thu, Jun 18, 2020 at 1:06 PM Mikhail Khludnev <[email protected]> > wrote: > > Hello, > > I'm challenged with cluster recovery. Think about total failure: ZK > state is lost, however instanceDirs survived since they are mounted via > EBS. Let's say collection is read/only and/or it doesn't have replicas, > just leaders. > > Is there a way to create a new empty collection and say, hey here's > shard1 instance, shard2 instance is there etc? > > > > Customer says that the old version of solr does it automatically: when > empty zk is connected, collection's shards just appear there. Right now due > to https://issues.apache.org/jira/browse/SOLR-12066 Cleanup deleted core > when node start - if instances with data dirs connect to empty ZK it just > wipes dirs away. > > > > Thanks > > -- > > Sincerely yours > > Mikhail Khludnev > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
