Is it correct to say that a lost node cannot be restored from backup? You must either replace the node or evict/re-add (i.e., rebuild from other nodes).
Also, that snapshot, incremental, commitlog backups are relegated to application keyspace recovery only? How about recovery of the entire cluster? (rolling it back). Are snapshots exact enough, in time, to not have a nodes that differ, in point-in-time, from the rest of the cluster? Would those nodes be recoverable (nodetool repair?) … which brings me back to recovering a lost node from backup (restore last snapshot, and run nodetool repair?). Thanks, Alan Gano From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Wednesday, June 12, 2019 10:14 AM To: user@cassandra.apache.org Subject: Re: Recover lost node from backup or evict/re-add? A host can replace itself using the method I described On Jun 12, 2019, at 7:10 AM, Alan Gano <ag...@tsys.com<mailto:ag...@tsys.com>> wrote: I guess I’m considering this scenario: · host and configuration have survived · /data is gone · /backups have survived I have tested recovering from this scenario with an evict/re-add, which worked fine. If I restore from backup, the node will be behind the cluster – errrr, does it get caught up after a restore and start it up? Alan From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Wednesday, June 12, 2019 10:02 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Recover lost node from backup or evict/re-add? To avoid violating consistency guarantees, you have to repair the replicas while the lost node is down Once you do that it’s typically easiest to bootstrap a replacement (there’s a property named “replace address first boot” you can google or someone can link) that tells a new joining host to take over for a failed machine. On Jun 12, 2019, at 6:54 AM, Alan Gano <ag...@tsys.com<mailto:ag...@tsys.com>> wrote: If I lose a node, does it make sense to even restore from snapshot/incrementals/commitlogs? Or is the best way to do an evict/re-add? Thanks, Alan. NOTICE: This communication is intended only for the person or entity to whom it is addressed and may contain confidential, proprietary, and/or privileged material. Unless you are the intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever of this communication is strictly prohibited. If you received this in error, please reply immediately and delete the material from all computers. Email sent through the Internet is not secure. Do not use email to send us confidential information such as credit card numbers, PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential information. NOTICE: This communication is intended only for the person or entity to whom it is addressed and may contain confidential, proprietary, and/or privileged material. Unless you are the intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever of this communication is strictly prohibited. If you received this in error, please reply immediately and delete the material from all computers. Email sent through the Internet is not secure. Do not use email to send us confidential information such as credit card numbers, PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential information. NOTICE: This communication is intended only for the person or entity to whom it is addressed and may contain confidential, proprietary, and/or privileged material. Unless you are the intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever of this communication is strictly prohibited. If you received this in error, please reply immediately and delete the material from all computers. Email sent through the Internet is not secure. Do not use email to send us confidential information such as credit card numbers, PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential information.
