Re: Disconnecting two data centers
On 2017-03-08 07:57 (-0800), Chuck Reynolds wrote: > I was hoping I could do the following > > · Change seeds Definitely. > > · Change the topology back to simply > Not necessary, can just remove the "other" datacenter from the replication strategy. > · Stop nodes in datacenter 2 > > · Remove nodes in datacenter 2 > >From where? system.peers? > · Restart nodes in datacenter 2 Then dc2 connects back to dc1 and you end up messed up again. BOTH SIDES are going to try to reconnect, and it's going to be ugly. If you used an internode authenticator, you could make that lock out the other cluster. If you used a property file snitch (NOT GPFS, plain old PFS), you could remove the other datacenter from each topology file. You can use a firewall. You'll defintiely need to change seeds, and probably need to stop most of the nodes all at once to enable this. > > Somehow Cassandra holds on to the information about who was in the cluster. > system.peers keeps a list of all members in the cluster. System is local strategy (not replicated), so each node has its own copy, so IF you try hacking at it, you MUST hack at it on all nodes (basically at the same time, because they'll repopulate via gossip). > What if I also changed the cluster name in the Cassandra.yaml before > restarting? > Changing the name is very difficult (decidedly nontrivial). Given that you need to stop all the nodes to do this (downtime), I'd be pretty tempted to tell you to nuke one of the clusters, and use sstableloader to repopulate it as a brand new cluster after you've nuked it. That is - Don't try to split a cluster in half, create it, then kill half, then make a new cluster, and use sstableloader to repopulate it quickly.
Re: Disconnecting two data centers
Do not change the cluster name - the cassandra service will not start on the same sstables if the cluster name is changed. Arvydas On Wed, Mar 8, 2017 at 4:57 PM, Chuck Reynolds wrote: > I was hoping I could do the following > > · Change seeds > > · Change the topology back to simply > > · Stop nodes in datacenter 2 > > · Remove nodes in datacenter 2 > > · Restart nodes in datacenter 2 > > > > Somehow Cassandra holds on to the information about who was in the cluster. > > > > What if I also changed the cluster name in the Cassandra.yaml before > restarting? > > > > Is the data bound to the cluster name? > > > > *From: *Ryan Svihla > *Reply-To: *"user@cassandra.apache.org" > *Date: *Wednesday, March 8, 2017 at 7:38 AM > *To: *user > *Subject: *Re: Disconnecting two data centers > > > > it's a bit tricky and I don't advise it, but the typical pattern is (say > you have DC1 and DC2): > > > > 1. partition the data centers from one another..kill the routing however > you can (firewall, etc) > > 2. while partitioned log onto DC1 alter schema so that DC2 is not > replicating), repeat for other. > > 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property > files and vice versa > > 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's > shouldn't have any seeds from DC2, etc) > > 3. rolling restart to account for this. > > 4,. run repair (not even sure how necessary this step is, but after doing > RF changes I do this to prevent hiccups) > > > > I've done this a couple of times but really failing all of that, the more > well supported and harder to mess up but more work approach is: > > > > 1. Set DC2 to RF 0 > > 2. remove all nodes from DC2 > > 3. change yamls for seed files (update property file if need be) > > 4. create new cluster in DC2, > > 5. use sstableloader to stream DC1 data to DC2. > > > > On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds > wrote: > > I’m running C* 2.1.13 and I have two rings that are replicating data from > our data center to one in AWS. > > > > We would like to keep both of them for a while but we have a need to > disconnect them. How can this be done? > > > > > > -- > > Thanks, > > Ryan Svihla >
Re: Disconnecting two data centers
I guess it depends on the experience one has. This is a common process to bring up, move, build full prod copies, etc. What is outlined is pretty much exactly what I have done 20-50 times (too many to remember). FYI, some of this should be done with nodes DOWN. *...* *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Mar 8, 2017 at 6:38 AM, Ryan Svihla wrote: > it's a bit tricky and I don't advise it, but the typical pattern is (say > you have DC1 and DC2): > > 1. partition the data centers from one another..kill the routing however > you can (firewall, etc) > 2. while partitioned log onto DC1 alter schema so that DC2 is not > replicating), repeat for other. > 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property > files and vice versa > 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's > shouldn't have any seeds from DC2, etc) > 3. rolling restart to account for this. > 4,. run repair (not even sure how necessary this step is, but after doing > RF changes I do this to prevent hiccups) > > I've done this a couple of times but really failing all of that, the more > well supported and harder to mess up but more work approach is: > > 1. Set DC2 to RF 0 > 2. remove all nodes from DC2 > 3. change yamls for seed files (update property file if need be) > 4. create new cluster in DC2, > 5. use sstableloader to stream DC1 data to DC2. > > On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds > wrote: > >> I’m running C* 2.1.13 and I have two rings that are replicating data from >> our data center to one in AWS. >> >> >> >> We would like to keep both of them for a while but we have a need to >> disconnect them. How can this be done? >> > > > > -- > > Thanks, > Ryan Svihla > >
Re: Is it possible to recover a deleted-in-future record?
Those future tombstones are going to continue to cause problems on those partitions. If you're still writing to those partitions, you might be losing data in the mean time. It's going to be hard to get the tombstone out of the way so that new writes can begin to happen there (newly written data will be occluded by the existing tombstones). Manual cleanup might be required here, such sstablefilter or sstable2json->clean up the data -> json2sstable. This could get really hairy. Another option, depending on the kind of tombstone they were (eg cell level), my deleting compactor[1] might be able to clean them up on the live cluster via user defined compaction if you wrote a convictor for this purpose. But that tool has a gap for cluster and/or partition level tombstones which it doesn't properly recognize yet (there's an open PR that provides partial implementation, but I'm not sure it would get you what you need). You can see my talk about that[2]. Careful caveat on this though, the deleting compactor was written to _avoid_ tombstones, it hasn't been well tested against data that contains tombstones, so although time is critical for you here to avoid ongoing corruption of your data while those bad tombstones remain in the way, I would still fully encourage you to validate whether this could satisfy your use case. [1] https://github.com/protectwise/cassandra-util [2] https://www.youtube.com/watch?v=BhGkSnBZgJA On Wed, Mar 8, 2017 at 6:06 AM Arvydas Jonusonis < arvydas.jonuso...@gmail.com> wrote: > That's a good point - a snapshot is certainly in order ASAP, if not > already done. > > One more thing I'd add about "data has to be consolidated from all the > nodes" (from #3 below): > >- EITHER run the sstable2json ops on each node >- OR if size permits, copy the relevant sstables (containing the >desired keys, from the output of the nodetool getsstables) locally or onto >a new single-node instance, start that instance and run the commands there > > If restoring the sstables from a snapshot, you'll need to do the latter > anyway. > > Arvydas > > On Wed, Mar 8, 2017 at 1:55 PM, Anuj Wadehra > wrote: > > DISCLAIMER: This is only my personal opinion. Evaluate the situation > carefully and if you find below suggestions useful, follow them at your own > risk. > > If I have understood the problem correctly, malicious deletes would > actually lead to deletion of data. I am not sure how everything is normal > after the deletes? > > If data is critical,you could: > > 1. Take a database snapshot immediately so that you dont lose information > if delete entrues in sstables are compacted together with original data. > > 2. Transfer snapshot to suitable place and Run some utility such as > sstabletojson to get the keys impacted by the deletes and original data for > keys. Data has to be consolidated from all the nodes. > > 3. Devise a strategy to restore deleted data. > > Thanks > Anuj > > > > On Tue, Mar 7, 2017 at 8:44 AM, Michael Fong > wrote: > > Hi, all, > > > > > > We recently encountered an issue in production that some records were > mysteriously deleted with a timestamp 100+ years from now. Everything is > normal as of now, and how the deletion happened and accuracy of system > timestamp at that moment are unknown. We were wondering if there is a > general way to recover the mysteriously-deleted data when the timestamp > meta is screwed up. > > > > Thanks in advanced, > > > > Regards, > > > > Michael Fong > > >
Re: Disconnecting two data centers
I was hoping I could do the following · Change seeds · Change the topology back to simply · Stop nodes in datacenter 2 · Remove nodes in datacenter 2 · Restart nodes in datacenter 2 Somehow Cassandra holds on to the information about who was in the cluster. What if I also changed the cluster name in the Cassandra.yaml before restarting? Is the data bound to the cluster name? From: Ryan Svihla Reply-To: "user@cassandra.apache.org" Date: Wednesday, March 8, 2017 at 7:38 AM To: user Subject: Re: Disconnecting two data centers it's a bit tricky and I don't advise it, but the typical pattern is (say you have DC1 and DC2): 1. partition the data centers from one another..kill the routing however you can (firewall, etc) 2. while partitioned log onto DC1 alter schema so that DC2 is not replicating), repeat for other. 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property files and vice versa 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's shouldn't have any seeds from DC2, etc) 3. rolling restart to account for this. 4,. run repair (not even sure how necessary this step is, but after doing RF changes I do this to prevent hiccups) I've done this a couple of times but really failing all of that, the more well supported and harder to mess up but more work approach is: 1. Set DC2 to RF 0 2. remove all nodes from DC2 3. change yamls for seed files (update property file if need be) 4. create new cluster in DC2, 5. use sstableloader to stream DC1 data to DC2. On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds mailto:creyno...@ancestry.com>> wrote: I’m running C* 2.1.13 and I have two rings that are replicating data from our data center to one in AWS. We would like to keep both of them for a while but we have a need to disconnect them. How can this be done? -- Thanks, Ryan Svihla
Re: Disconnecting two data centers
it's a bit tricky and I don't advise it, but the typical pattern is (say you have DC1 and DC2): 1. partition the data centers from one another..kill the routing however you can (firewall, etc) 2. while partitioned log onto DC1 alter schema so that DC2 is not replicating), repeat for other. 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property files and vice versa 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's shouldn't have any seeds from DC2, etc) 3. rolling restart to account for this. 4,. run repair (not even sure how necessary this step is, but after doing RF changes I do this to prevent hiccups) I've done this a couple of times but really failing all of that, the more well supported and harder to mess up but more work approach is: 1. Set DC2 to RF 0 2. remove all nodes from DC2 3. change yamls for seed files (update property file if need be) 4. create new cluster in DC2, 5. use sstableloader to stream DC1 data to DC2. On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds wrote: > I’m running C* 2.1.13 and I have two rings that are replicating data from > our data center to one in AWS. > > > > We would like to keep both of them for a while but we have a need to > disconnect them. How can this be done? > -- Thanks, Ryan Svihla
Disconnecting two data centers
I’m running C* 2.1.13 and I have two rings that are replicating data from our data center to one in AWS. We would like to keep both of them for a while but we have a need to disconnect them. How can this be done?
Re: Is it possible to recover a deleted-in-future record?
That's a good point - a snapshot is certainly in order ASAP, if not already done. One more thing I'd add about "data has to be consolidated from all the nodes" (from #3 below): - EITHER run the sstable2json ops on each node - OR if size permits, copy the relevant sstables (containing the desired keys, from the output of the nodetool getsstables) locally or onto a new single-node instance, start that instance and run the commands there If restoring the sstables from a snapshot, you'll need to do the latter anyway. Arvydas On Wed, Mar 8, 2017 at 1:55 PM, Anuj Wadehra wrote: > DISCLAIMER: This is only my personal opinion. Evaluate the situation > carefully and if you find below suggestions useful, follow them at your own > risk. > > If I have understood the problem correctly, malicious deletes would > actually lead to deletion of data. I am not sure how everything is normal > after the deletes? > > If data is critical,you could: > > 1. Take a database snapshot immediately so that you dont lose information > if delete entrues in sstables are compacted together with original data. > > 2. Transfer snapshot to suitable place and Run some utility such as > sstabletojson to get the keys impacted by the deletes and original data for > keys. Data has to be consolidated from all the nodes. > > 3. Devise a strategy to restore deleted data. > > Thanks > Anuj > > > > On Tue, Mar 7, 2017 at 8:44 AM, Michael Fong > wrote: > > Hi, all, > > > > > > We recently encountered an issue in production that some records were > mysteriously deleted with a timestamp 100+ years from now. Everything is > normal as of now, and how the deletion happened and accuracy of system > timestamp at that moment are unknown. We were wondering if there is a > general way to recover the mysteriously-deleted data when the timestamp > meta is screwed up. > > > > Thanks in advanced, > > > > Regards, > > > > Michael Fong > >
Re: Is it possible to recover a deleted-in-future record?
DISCLAIMER: This is only my personal opinion. Evaluate the situation carefully and if you find below suggestions useful, follow them at your own risk. If I have understood the problem correctly, malicious deletes would actually lead to deletion of data. I am not sure how everything is normal after the deletes? If data is critical,you could: 1. Take a database snapshot immediately so that you dont lose information if delete entrues in sstables are compacted together with original data. 2. Transfer snapshot to suitable place and Run some utility such as sstabletojson to get the keys impacted by the deletes and original data for keys. Data has to be consolidated from all the nodes. 3. Devise a strategy to restore deleted data. ThanksAnuj On Tue, Mar 7, 2017 at 8:44 AM, Michael Fong wrote: Hi, all, We recently encountered an issue in production that some records were mysteriously deleted with a timestamp 100+ years from now. Everything is normal as of now, and how the deletion happened and accuracy of system timestamp at that moment are unknown. We were wondering if there is a general way to recover the mysteriously-deleted data when the timestamp meta is screwed up. Thanks in advanced, Regards, Michael Fong
Re: Can I do point in time recover using nodetool
Yes, It's possible. I haven't seen good instructions online though. The Cassandra docs are quite bad as well. I think I asked about it in this list and therefore I suggest you check the mailing list archive as Mr. Roth suggested. Hannu On Wed, 8 Mar 2017 at 10.50, benjamin roth wrote: > I remember a very similar question on the list some months ago. > The short answer is that there is no short answer. I'd recommend you > search the mailing list archive for "backup" or "recover". > > 2017-03-08 10:17 GMT+01:00 Bhardwaj, Rahul : > > Hi All, > > > > Is there any possibility of restoring cassandra snapshots to point in time > without using opscenter ? > > > > > > > > > > *Thanks and Regards* > > *Rahul Bhardwaj* > > > > >
Re: Is it possible to recover a deleted-in-future record?
Use nodetool getsstables to discover which sstables contain the data and then dump it with sstable2json -k to explore the content of the data/mutations for those keys. Arvydas On Tue, Mar 7, 2017 at 4:13 AM, Michael Fong < michael.f...@ruckuswireless.com> wrote: > Hi, all, > > > > > > We recently encountered an issue in production that some records were > mysteriously deleted with a timestamp 100+ years from now. Everything is > normal as of now, and how the deletion happened and accuracy of system > timestamp at that moment are unknown. We were wondering if there is a > general way to recover the mysteriously-deleted data when the timestamp > meta is screwed up. > > > > Thanks in advanced, > > > > Regards, > > > > Michael Fong >
Re: Can I do point in time recover using nodetool
I remember a very similar question on the list some months ago. The short answer is that there is no short answer. I'd recommend you search the mailing list archive for "backup" or "recover". 2017-03-08 10:17 GMT+01:00 Bhardwaj, Rahul : > Hi All, > > > > Is there any possibility of restoring cassandra snapshots to point in time > without using opscenter ? > > > > > > > > > > *Thanks and Regards* > > *Rahul Bhardwaj* > > >
Can I do point in time recover using nodetool
Hi All, Is there any possibility of restoring cassandra snapshots to point in time without using opscenter ? Thanks and Regards Rahul Bhardwaj
