I’m not sure it is correct to say, “you cannot.” However, that is a more complicated restore and more likely to lead to inconsistent data and take longer to do. You are basically trying to start from a backup point and roll everything forward and catch up to current.
Replacing/re-streaming is the well-trodden path. You are getting the net result of all that has happened since the node failure. And the node is not returning data to the clients while the bootstrap is running. If you have a restored/repairing node, it will accept client (and coordinator) connections even though it isn’t (guaranteed) consistent, yet. As I understand it – a full cluster recovery from backup still requires repair across the cluster to ensure consistency. In my experience, most apps cannot wait for a full restore/repair. Availability matters more. They also don’t want to pay for even more disk to hold some level of backups. There are some companies that provide finer-grained backup and recovery options, though. Sean Durity From: Alan Gano <ag...@tsys.com> Sent: Wednesday, June 12, 2019 1:43 PM To: user@cassandra.apache.org Subject: [EXTERNAL] RE: Recover lost node from backup or evict/re-add? Is it correct to say that a lost node cannot be restored from backup? You must either replace the node or evict/re-add (i.e., rebuild from other nodes). Also, that snapshot, incremental, commitlog backups are relegated to application keyspace recovery only? How about recovery of the entire cluster? (rolling it back). Are snapshots exact enough, in time, to not have a nodes that differ, in point-in-time, from the rest of the cluster? Would those nodes be recoverable (nodetool repair?) … which brings me back to recovering a lost node from backup (restore last snapshot, and run nodetool repair?). Thanks, Alan Gano From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Wednesday, June 12, 2019 10:14 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Recover lost node from backup or evict/re-add? A host can replace itself using the method I described On Jun 12, 2019, at 7:10 AM, Alan Gano <ag...@tsys.com<mailto:ag...@tsys.com>> wrote: I guess I’m considering this scenario: * host and configuration have survived * /data is gone * /backups have survived I have tested recovering from this scenario with an evict/re-add, which worked fine. If I restore from backup, the node will be behind the cluster – errrr, does it get caught up after a restore and start it up? Alan From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Wednesday, June 12, 2019 10:02 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Recover lost node from backup or evict/re-add? To avoid violating consistency guarantees, you have to repair the replicas while the lost node is down Once you do that it’s typically easiest to bootstrap a replacement (there’s a property named “replace address first boot” you can google or someone can link) that tells a new joining host to take over for a failed machine. On Jun 12, 2019, at 6:54 AM, Alan Gano <ag...@tsys.com<mailto:ag...@tsys.com>> wrote: If I lose a node, does it make sense to even restore from snapshot/incrementals/commitlogs? Or is the best way to do an evict/re-add? Thanks, Alan. NOTICE: This communication is intended only for the person or entity to whom it is addressed and may contain confidential, proprietary, and/or privileged material. Unless you are the intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever of this communication is strictly prohibited. If you received this in error, please reply immediately and delete the material from all computers. Email sent through the Internet is not secure. Do not use email to send us confidential information such as credit card numbers, PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential information. NOTICE: This communication is intended only for the person or entity to whom it is addressed and may contain confidential, proprietary, and/or privileged material. Unless you are the intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever of this communication is strictly prohibited. If you received this in error, please reply immediately and delete the material from all computers. Email sent through the Internet is not secure. Do not use email to send us confidential information such as credit card numbers, PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential information. NOTICE: This communication is intended only for the person or entity to whom it is addressed and may contain confidential, proprietary, and/or privileged material. Unless you are the intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever of this communication is strictly prohibited. If you received this in error, please reply immediately and delete the material from all computers. Email sent through the Internet is not secure. Do not use email to send us confidential information such as credit card numbers, PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential information. ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
