100% agree with Sean. I would only use Cassandra backups in a case where you need to restore from full cluster loss. Example: An entire DC burns down, tornado, flooding.
Your routine node replacement after a failure should be replace_address_first_boot. To ensure this goes smoothly, run regular repairs. We (The Last Pickle) maintain this to make it easy: http://cassandra-reaper.io/ Jon On Wed, Jun 12, 2019 at 11:17 AM Durity, Sean R <sean_r_dur...@homedepot.com> wrote: > I’m not sure it is correct to say, “you cannot.” However, that is a more > complicated restore and more likely to lead to inconsistent data and take > longer to do. You are basically trying to start from a backup point and > roll everything forward and catch up to current. > > > > Replacing/re-streaming is the well-trodden path. You are getting the net > result of all that has happened since the node failure. And the node is not > returning data to the clients while the bootstrap is running. If you have a > restored/repairing node, it will accept client (and coordinator) > connections even though it isn’t (guaranteed) consistent, yet. > > > > As I understand it – a full cluster recovery from backup still requires > repair across the cluster to ensure consistency. In my experience, most > apps cannot wait for a full restore/repair. Availability matters more. They > also don’t want to pay for even more disk to hold some level of backups. > > > > There are some companies that provide finer-grained backup and recovery > options, though. > > > > Sean Durity > > > > *From:* Alan Gano <ag...@tsys.com> > *Sent:* Wednesday, June 12, 2019 1:43 PM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] RE: Recover lost node from backup or evict/re-add? > > > > > > Is it correct to say that a lost node cannot be restored from backup? You > must either replace the node or evict/re-add (i.e., rebuild from other > nodes). > > > > Also, that snapshot, incremental, commitlog backups are relegated to > application keyspace recovery only? > > > > > > How about recovery of the entire cluster? (rolling it back). Are > snapshots exact enough, in time, to not have a nodes that differ, in > point-in-time, from the rest of the cluster? Would those nodes be > recoverable (nodetool repair?) … which brings me back to recovering a lost > node from backup (restore last snapshot, and run nodetool repair?). > > > > > > Thanks, > > > > Alan Gano > > > > > > *From:* Jeff Jirsa [mailto:jji...@gmail.com <jji...@gmail.com>] > *Sent:* Wednesday, June 12, 2019 10:14 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Recover lost node from backup or evict/re-add? > > > > A host can replace itself using the method I described > > > On Jun 12, 2019, at 7:10 AM, Alan Gano <ag...@tsys.com> wrote: > > I guess I’m considering this scenario: > > - host and configuration have survived > - /data is gone > - /backups have survived > > > > I have tested recovering from this scenario with an evict/re-add, which > worked fine. > > > > If I restore from backup, the node will be behind the cluster – errrr, > does it get caught up after a restore and start it up? > > > > Alan > > > > *From:* Jeff Jirsa [mailto:jji...@gmail.com <jji...@gmail.com>] > *Sent:* Wednesday, June 12, 2019 10:02 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Recover lost node from backup or evict/re-add? > > > > To avoid violating consistency guarantees, you have to repair the replicas > while the lost node is down > > > > Once you do that it’s typically easiest to bootstrap a replacement > (there’s a property named “replace address first boot” you can google or > someone can link) that tells a new joining host to take over for a failed > machine. > > > > > On Jun 12, 2019, at 6:54 AM, Alan Gano <ag...@tsys.com> wrote: > > > > If I lose a node, does it make sense to even restore from > snapshot/incrementals/commitlogs? > > > > Or is the best way to do an evict/re-add? > > > > > > Thanks, > > > > Alan. > > > > NOTICE: This communication is intended only for the person or entity to > whom it is addressed and may contain confidential, proprietary, and/or > privileged material. Unless you are the intended addressee, any review, > reliance, dissemination, distribution, copying or use whatsoever of this > communication is strictly prohibited. If you received this in error, please > reply immediately and delete the material from all computers. Email sent > through the Internet is not secure. Do not use email to send us > confidential information such as credit card numbers, PIN numbers, > passwords, Social Security Numbers, Account numbers, or other important and > confidential information. > > NOTICE: This communication is intended only for the person or entity to > whom it is addressed and may contain confidential, proprietary, and/or > privileged material. Unless you are the intended addressee, any review, > reliance, dissemination, distribution, copying or use whatsoever of this > communication is strictly prohibited. If you received this in error, please > reply immediately and delete the material from all computers. Email sent > through the Internet is not secure. Do not use email to send us > confidential information such as credit card numbers, PIN numbers, > passwords, Social Security Numbers, Account numbers, or other important and > confidential information. > > NOTICE: This communication is intended only for the person or entity to > whom it is addressed and may contain confidential, proprietary, and/or > privileged material. Unless you are the intended addressee, any review, > reliance, dissemination, distribution, copying or use whatsoever of this > communication is strictly prohibited. If you received this in error, please > reply immediately and delete the material from all computers. Email sent > through the Internet is not secure. Do not use email to send us > confidential information such as credit card numbers, PIN numbers, > passwords, Social Security Numbers, Account numbers, or other important and > confidential information. > > ------------------------------ > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
