Re: Adding nodes to existing cluster
OK. Thanks. I'll monitor the resources status (network, memory, cpu, io) as I go and try to bootsrap them at chunks which seems not to have a bad impact. Will do regarding the cleanup. Thanks! On Mon, Apr 20, 2015 at 4:08 PM, Carlos Rolo wrote: > Independent of the snitch, data needs to travel to the new nodes (plus all > the keyspace information that goes via gossip). So I won't bootstrap them > all at once, even if it is only for network traffic generated. > > Don't forget to run cleanup on the old nodes once all nodes are in place to > reclaim disk space. > > Regards, > > Carlos Juzarte Rolo > Cassandra Consultant > > Pythian - Love your data > > rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 > www.pythian.com > > On Mon, Apr 20, 2015 at 1:58 PM, Or Sher wrote: >> >> Thanks for the response. >> Sure we'll monitor as we're adding nodes. >> We're now using 6 nodes on each DC. (We have 2 DCs) >> Each node contains ~800GB >> >> Do you know how rack configurations are relevant here? >> Do you see any reason to bootstrap them one by one if we're not using >> rack awareness? >> >> >> On Mon, Apr 20, 2015 at 2:49 PM, Carlos Rolo wrote: >> > Start one node at a time. Wait 2 minutes before starting each node. >> > >> > >> > How much data and nodes you have already? Depending on that, the >> > streaming >> > of data can stress on the resources you have. >> > I would recommend to start one and monitor, if things are ok, add >> > another >> > one. And so on. >> > >> > Regards, >> > >> > Carlos Juzarte Rolo >> > Cassandra Consultant >> > >> > Pythian - Love your data >> > >> > rolo@pythian | Twitter: cjrolo | Linkedin: >> > linkedin.com/in/carlosjuzarterolo >> > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 >> > www.pythian.com >> > >> > On Mon, Apr 20, 2015 at 11:02 AM, Or Sher wrote: >> >> >> >> Hi all, >> >> In the near future I'll need to add more than 10 nodes to a 2.0.9 >> >> cluster (using vnodes). >> >> I read this documentation on datastax website: >> >> >> >> >> >> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html >> >> >> >> In one point it says: >> >> "If you are using racks, you can safely bootstrap two nodes at a time >> >> when both nodes are on the same rack." >> >> >> >> And in another is says: >> >> "Start Cassandra on each new node. Allow two minutes between node >> >> initializations. You can monitor the startup and data streaming >> >> process using nodetool netstats." >> >> >> >> We're not using racks configuration and from reading this >> >> documentation I'm not really sure is it safe for us to bootstrap all >> >> nodes together (with two minutes between each other). >> >> I really hate the tought of doing it one by one, I assume it will take >> >> more than 6H per node. >> >> >> >> What do you say? >> >> -- >> >> Or Sher >> > >> > >> > >> > -- >> > >> > >> > >> >> >> >> -- >> Or Sher > > > > -- > > > -- Or Sher
Re: Adding nodes to existing cluster
Thanks for the response. Sure we'll monitor as we're adding nodes. We're now using 6 nodes on each DC. (We have 2 DCs) Each node contains ~800GB Do you know how rack configurations are relevant here? Do you see any reason to bootstrap them one by one if we're not using rack awareness? On Mon, Apr 20, 2015 at 2:49 PM, Carlos Rolo wrote: > Start one node at a time. Wait 2 minutes before starting each node. > > > How much data and nodes you have already? Depending on that, the streaming > of data can stress on the resources you have. > I would recommend to start one and monitor, if things are ok, add another > one. And so on. > > Regards, > > Carlos Juzarte Rolo > Cassandra Consultant > > Pythian - Love your data > > rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 > www.pythian.com > > On Mon, Apr 20, 2015 at 11:02 AM, Or Sher wrote: >> >> Hi all, >> In the near future I'll need to add more than 10 nodes to a 2.0.9 >> cluster (using vnodes). >> I read this documentation on datastax website: >> >> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html >> >> In one point it says: >> "If you are using racks, you can safely bootstrap two nodes at a time >> when both nodes are on the same rack." >> >> And in another is says: >> "Start Cassandra on each new node. Allow two minutes between node >> initializations. You can monitor the startup and data streaming >> process using nodetool netstats." >> >> We're not using racks configuration and from reading this >> documentation I'm not really sure is it safe for us to bootstrap all >> nodes together (with two minutes between each other). >> I really hate the tought of doing it one by one, I assume it will take >> more than 6H per node. >> >> What do you say? >> -- >> Or Sher > > > > -- > > > -- Or Sher
Adding nodes to existing cluster
Hi all, In the near future I'll need to add more than 10 nodes to a 2.0.9 cluster (using vnodes). I read this documentation on datastax website: http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html In one point it says: "If you are using racks, you can safely bootstrap two nodes at a time when both nodes are on the same rack." And in another is says: "Start Cassandra on each new node. Allow two minutes between node initializations. You can monitor the startup and data streaming process using nodetool netstats." We're not using racks configuration and from reading this documentation I'm not really sure is it safe for us to bootstrap all nodes together (with two minutes between each other). I really hate the tought of doing it one by one, I assume it will take more than 6H per node. What do you say? -- Or Sher
Re: Replacing nodes disks
Thanks guys. I think I'll start with the replacement of a dead node procedure at least for the first node and I'll monitor the cluster overhead and timing. If I'll see that the overhead and elapsed time are substantially higher I'll try to find some network storage to store the backup. Using the replacement procedure will also be a great practice on production size data, and it will be less dangerous as it's on DR nodes. On Mon, Dec 22, 2014 at 6:22 PM, Jan Kesten wrote: > Hi, > > even if recovery like a dead node would work - backup and restore (like my > way with an usb docking station) will be much faster and produce less IO > and CPU impact on your cluster. > > Keep that in Mind :-) > > Cheers, > Jan > > Am 22.12.2014 um 10:58 schrieb Or Sher: > > Great. replace_address works great. > From some reason I thought it won't work with the same IP. > > > On Sun, Dec 21, 2014 at 5:14 PM, Ryan Svihla wrote: > >> Cassandra is designed to rebuild a node from other nodes, whether a node >> is dead by your hand because you killed it or fate is irrelevant, the >> process is the same, a "new node" can be the same hostname and ip or it can >> have totally different ones. >> >> On Sun, Dec 21, 2014 at 6:01 AM, Or Sher wrote: >>> >>> If I'll use the replace_address parameter with the same IP address, >>> would that do the job? >>> >>> On Sun, Dec 21, 2014 at 11:20 AM, Or Sher wrote: >>> >>>> What I want to do is kind of replacing a dead node - >>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html >>>> But replacing it with a clean node with the same IP and hostname. >>>> >>>> On Sun, Dec 21, 2014 at 9:53 AM, Or Sher wrote: >>>> >>>>> Thanks guys. >>>>> I have to replace all data disks, so I don't have another large enough >>>>> local disk to move the data to. >>>>> If I'll have no choice, I will backup the data before on some other >>>>> node or something, but I'd like to avoid it. >>>>> I would really love letting Cassandra do it thing and rebuild itself. >>>>> Did anybody handled such cases that way (Letting Cassandra rebuild >>>>> it's data?) >>>>> Although there are no documented procedure for it, It should be >>>>> possible right? >>>>> >>>>> On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten >>>>> wrote: >>>>> >>>>>> Hi Or, >>>>>> >>>>>> I did some sort of this a while ago. If your machines do have a free >>>>>> disk slot - just put another disk there and use it as another >>>>>> data_file_directory. >>>>>> >>>>>> If not - as in my case: >>>>>> >>>>>> - grab an usb dock for disks >>>>>> - put the new one in there, plug in, format, mount to /mnt etc. >>>>>> - I did an online rsync from /var/lib/cassandra/data to /mnt >>>>>> - after that, bring cassandra down >>>>>> - do another rsync from /var/lib/cassandra/data to /mnt (should be >>>>>> faster, as sstables do not change, minimizes downtime) >>>>>> - if you need adjust /etc/fstab if needed >>>>>> - shutdown the node >>>>>> - swap disks >>>>>> - power on the node >>>>>> - everything should be fine ;-) >>>>>> >>>>>> Of course you will need a replication factor > 1 for this to work ;-) >>>>>> >>>>>> Just my 2 cents, >>>>>> Jan >>>>>> >>>>>> rsync the full contents there, >>>>>> >>>>>> Am 18.12.2014 um 16:17 schrieb Or Sher: >>>>>> >>>>>> Hi all, >>>>>>> >>>>>>> We have a situation where some of our nodes have smaller disks and >>>>>>> we would like to align all nodes by replacing the smaller disks to >>>>>>> bigger >>>>>>> ones without replacing nodes. >>>>>>> We don't have enough space to put data on / disk and copy it back to >>>>>>> the bigger disks so we would like to rebuild the nodes data from other >>>>>>> replicas. >>>>>>> >>>>>>> What do you think should be the procedure here? &
Re: Replacing nodes disks
Great. replace_address works great. >From some reason I thought it won't work with the same IP. On Sun, Dec 21, 2014 at 5:14 PM, Ryan Svihla wrote: > Cassandra is designed to rebuild a node from other nodes, whether a node > is dead by your hand because you killed it or fate is irrelevant, the > process is the same, a "new node" can be the same hostname and ip or it can > have totally different ones. > > On Sun, Dec 21, 2014 at 6:01 AM, Or Sher wrote: >> >> If I'll use the replace_address parameter with the same IP address, >> would that do the job? >> >> On Sun, Dec 21, 2014 at 11:20 AM, Or Sher wrote: >> >>> What I want to do is kind of replacing a dead node - >>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html >>> But replacing it with a clean node with the same IP and hostname. >>> >>> On Sun, Dec 21, 2014 at 9:53 AM, Or Sher wrote: >>> >>>> Thanks guys. >>>> I have to replace all data disks, so I don't have another large enough >>>> local disk to move the data to. >>>> If I'll have no choice, I will backup the data before on some other >>>> node or something, but I'd like to avoid it. >>>> I would really love letting Cassandra do it thing and rebuild itself. >>>> Did anybody handled such cases that way (Letting Cassandra rebuild it's >>>> data?) >>>> Although there are no documented procedure for it, It should be >>>> possible right? >>>> >>>> On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten >>>> wrote: >>>> >>>>> Hi Or, >>>>> >>>>> I did some sort of this a while ago. If your machines do have a free >>>>> disk slot - just put another disk there and use it as another >>>>> data_file_directory. >>>>> >>>>> If not - as in my case: >>>>> >>>>> - grab an usb dock for disks >>>>> - put the new one in there, plug in, format, mount to /mnt etc. >>>>> - I did an online rsync from /var/lib/cassandra/data to /mnt >>>>> - after that, bring cassandra down >>>>> - do another rsync from /var/lib/cassandra/data to /mnt (should be >>>>> faster, as sstables do not change, minimizes downtime) >>>>> - if you need adjust /etc/fstab if needed >>>>> - shutdown the node >>>>> - swap disks >>>>> - power on the node >>>>> - everything should be fine ;-) >>>>> >>>>> Of course you will need a replication factor > 1 for this to work ;-) >>>>> >>>>> Just my 2 cents, >>>>> Jan >>>>> >>>>> rsync the full contents there, >>>>> >>>>> Am 18.12.2014 um 16:17 schrieb Or Sher: >>>>> >>>>> Hi all, >>>>>> >>>>>> We have a situation where some of our nodes have smaller disks and we >>>>>> would like to align all nodes by replacing the smaller disks to bigger >>>>>> ones >>>>>> without replacing nodes. >>>>>> We don't have enough space to put data on / disk and copy it back to >>>>>> the bigger disks so we would like to rebuild the nodes data from other >>>>>> replicas. >>>>>> >>>>>> What do you think should be the procedure here? >>>>>> >>>>>> I'm guessing it should be something like this but I'm pretty sure >>>>>> it's not enough. >>>>>> 1. shutdown C* node and server. >>>>>> 2. replace disks + create the same vg lv etc. >>>>>> 3. start C* (Normally?) >>>>>> 4. nodetool repair/rebuild? >>>>>> *I think I might get some consistency issues for use cases relying on >>>>>> Quorum reads and writes for strong consistency. >>>>>> What do you say? >>>>>> >>>>>> Another question is (and I know it depends on many factors but I'd >>>>>> like to hear an experienced estimation): How much time would take to >>>>>> rebuild a 250G data node? >>>>>> >>>>>> Thanks in advance, >>>>>> Or. >>>>>> >>>>>> -- >>>>>> Or Sher >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Or Sher >>>> >>> >>> >>> >>> -- >>> Or Sher >>> >> >> >> >> -- >> Or Sher >> > > > -- > > [image: datastax_logo.png] <http://www.datastax.com/> > > Ryan Svihla > > Solution Architect > > [image: twitter.png] <https://twitter.com/foundev> [image: linkedin.png] > <http://www.linkedin.com/pub/ryan-svihla/12/621/727/> > > DataStax is the fastest, most scalable distributed database technology, > delivering Apache Cassandra to the world’s most innovative enterprises. > Datastax is built to be agile, always-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > -- Or Sher
Re: Replacing nodes disks
If I'll use the replace_address parameter with the same IP address, would that do the job? On Sun, Dec 21, 2014 at 11:20 AM, Or Sher wrote: > What I want to do is kind of replacing a dead node - > http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html > But replacing it with a clean node with the same IP and hostname. > > On Sun, Dec 21, 2014 at 9:53 AM, Or Sher wrote: > >> Thanks guys. >> I have to replace all data disks, so I don't have another large enough >> local disk to move the data to. >> If I'll have no choice, I will backup the data before on some other node >> or something, but I'd like to avoid it. >> I would really love letting Cassandra do it thing and rebuild itself. >> Did anybody handled such cases that way (Letting Cassandra rebuild it's >> data?) >> Although there are no documented procedure for it, It should be possible >> right? >> >> On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten wrote: >> >>> Hi Or, >>> >>> I did some sort of this a while ago. If your machines do have a free >>> disk slot - just put another disk there and use it as another >>> data_file_directory. >>> >>> If not - as in my case: >>> >>> - grab an usb dock for disks >>> - put the new one in there, plug in, format, mount to /mnt etc. >>> - I did an online rsync from /var/lib/cassandra/data to /mnt >>> - after that, bring cassandra down >>> - do another rsync from /var/lib/cassandra/data to /mnt (should be >>> faster, as sstables do not change, minimizes downtime) >>> - if you need adjust /etc/fstab if needed >>> - shutdown the node >>> - swap disks >>> - power on the node >>> - everything should be fine ;-) >>> >>> Of course you will need a replication factor > 1 for this to work ;-) >>> >>> Just my 2 cents, >>> Jan >>> >>> rsync the full contents there, >>> >>> Am 18.12.2014 um 16:17 schrieb Or Sher: >>> >>> Hi all, >>>> >>>> We have a situation where some of our nodes have smaller disks and we >>>> would like to align all nodes by replacing the smaller disks to bigger ones >>>> without replacing nodes. >>>> We don't have enough space to put data on / disk and copy it back to >>>> the bigger disks so we would like to rebuild the nodes data from other >>>> replicas. >>>> >>>> What do you think should be the procedure here? >>>> >>>> I'm guessing it should be something like this but I'm pretty sure it's >>>> not enough. >>>> 1. shutdown C* node and server. >>>> 2. replace disks + create the same vg lv etc. >>>> 3. start C* (Normally?) >>>> 4. nodetool repair/rebuild? >>>> *I think I might get some consistency issues for use cases relying on >>>> Quorum reads and writes for strong consistency. >>>> What do you say? >>>> >>>> Another question is (and I know it depends on many factors but I'd like >>>> to hear an experienced estimation): How much time would take to rebuild a >>>> 250G data node? >>>> >>>> Thanks in advance, >>>> Or. >>>> >>>> -- >>>> Or Sher >>>> >>> >>> >> >> >> -- >> Or Sher >> > > > > -- > Or Sher > -- Or Sher
Re: Replacing nodes disks
What I want to do is kind of replacing a dead node - http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html But replacing it with a clean node with the same IP and hostname. On Sun, Dec 21, 2014 at 9:53 AM, Or Sher wrote: > Thanks guys. > I have to replace all data disks, so I don't have another large enough > local disk to move the data to. > If I'll have no choice, I will backup the data before on some other node > or something, but I'd like to avoid it. > I would really love letting Cassandra do it thing and rebuild itself. > Did anybody handled such cases that way (Letting Cassandra rebuild it's > data?) > Although there are no documented procedure for it, It should be possible > right? > > On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten wrote: > >> Hi Or, >> >> I did some sort of this a while ago. If your machines do have a free disk >> slot - just put another disk there and use it as another >> data_file_directory. >> >> If not - as in my case: >> >> - grab an usb dock for disks >> - put the new one in there, plug in, format, mount to /mnt etc. >> - I did an online rsync from /var/lib/cassandra/data to /mnt >> - after that, bring cassandra down >> - do another rsync from /var/lib/cassandra/data to /mnt (should be >> faster, as sstables do not change, minimizes downtime) >> - if you need adjust /etc/fstab if needed >> - shutdown the node >> - swap disks >> - power on the node >> - everything should be fine ;-) >> >> Of course you will need a replication factor > 1 for this to work ;-) >> >> Just my 2 cents, >> Jan >> >> rsync the full contents there, >> >> Am 18.12.2014 um 16:17 schrieb Or Sher: >> >> Hi all, >>> >>> We have a situation where some of our nodes have smaller disks and we >>> would like to align all nodes by replacing the smaller disks to bigger ones >>> without replacing nodes. >>> We don't have enough space to put data on / disk and copy it back to the >>> bigger disks so we would like to rebuild the nodes data from other replicas. >>> >>> What do you think should be the procedure here? >>> >>> I'm guessing it should be something like this but I'm pretty sure it's >>> not enough. >>> 1. shutdown C* node and server. >>> 2. replace disks + create the same vg lv etc. >>> 3. start C* (Normally?) >>> 4. nodetool repair/rebuild? >>> *I think I might get some consistency issues for use cases relying on >>> Quorum reads and writes for strong consistency. >>> What do you say? >>> >>> Another question is (and I know it depends on many factors but I'd like >>> to hear an experienced estimation): How much time would take to rebuild a >>> 250G data node? >>> >>> Thanks in advance, >>> Or. >>> >>> -- >>> Or Sher >>> >> >> > > > -- > Or Sher > -- Or Sher
Re: Replacing nodes disks
Thanks guys. I have to replace all data disks, so I don't have another large enough local disk to move the data to. If I'll have no choice, I will backup the data before on some other node or something, but I'd like to avoid it. I would really love letting Cassandra do it thing and rebuild itself. Did anybody handled such cases that way (Letting Cassandra rebuild it's data?) Although there are no documented procedure for it, It should be possible right? On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten wrote: > Hi Or, > > I did some sort of this a while ago. If your machines do have a free disk > slot - just put another disk there and use it as another > data_file_directory. > > If not - as in my case: > > - grab an usb dock for disks > - put the new one in there, plug in, format, mount to /mnt etc. > - I did an online rsync from /var/lib/cassandra/data to /mnt > - after that, bring cassandra down > - do another rsync from /var/lib/cassandra/data to /mnt (should be faster, > as sstables do not change, minimizes downtime) > - if you need adjust /etc/fstab if needed > - shutdown the node > - swap disks > - power on the node > - everything should be fine ;-) > > Of course you will need a replication factor > 1 for this to work ;-) > > Just my 2 cents, > Jan > > rsync the full contents there, > > Am 18.12.2014 um 16:17 schrieb Or Sher: > > Hi all, >> >> We have a situation where some of our nodes have smaller disks and we >> would like to align all nodes by replacing the smaller disks to bigger ones >> without replacing nodes. >> We don't have enough space to put data on / disk and copy it back to the >> bigger disks so we would like to rebuild the nodes data from other replicas. >> >> What do you think should be the procedure here? >> >> I'm guessing it should be something like this but I'm pretty sure it's >> not enough. >> 1. shutdown C* node and server. >> 2. replace disks + create the same vg lv etc. >> 3. start C* (Normally?) >> 4. nodetool repair/rebuild? >> *I think I might get some consistency issues for use cases relying on >> Quorum reads and writes for strong consistency. >> What do you say? >> >> Another question is (and I know it depends on many factors but I'd like >> to hear an experienced estimation): How much time would take to rebuild a >> 250G data node? >> >> Thanks in advance, >> Or. >> >> -- >> Or Sher >> > > -- Or Sher
Replacing nodes disks
Hi all, We have a situation where some of our nodes have smaller disks and we would like to align all nodes by replacing the smaller disks to bigger ones without replacing nodes. We don't have enough space to put data on / disk and copy it back to the bigger disks so we would like to rebuild the nodes data from other replicas. What do you think should be the procedure here? I'm guessing it should be something like this but I'm pretty sure it's not enough. 1. shutdown C* node and server. 2. replace disks + create the same vg lv etc. 3. start C* (Normally?) 4. nodetool repair/rebuild? *I think I might get some consistency issues for use cases relying on Quorum reads and writes for strong consistency. What do you say? Another question is (and I know it depends on many factors but I'd like to hear an experienced estimation): How much time would take to rebuild a 250G data node? Thanks in advance, Or. -- Or Sher
Hector latency related configuration
Hi all, We're using Hector in one of our older use cases with C* 1.0.9. We suspect it increases our total round trip write latency to Cassandra. C* metrics shows low latency so we assume the problem is somewhere else. What are the configuration parameters you would recommend to investigate/change in order to decrease latency. -- Or Sher
Re: Cassandra process exiting mysteriously
Hi Clint, I think I kind of found the reason for my problem, I doubt you have the exact same problem but here it is: We're using Zabbix as our monitoring system and it uses /usr/bin/at to schedule it monitoring runs. Every time the "at" command adds another scheduled task, it send a kill signal to the pid of the atd, probably just to check if it's alive, not to kill it. Now, looking at the system calls audit log, it seems like sometimes, although the kill syscall uses one pid (the atd one), it actually send the kill to our C* java process. I'm really starting to think it's some kind of a linux kernel bug.. BTW, atd was always stopped, so I'm not really sure yet if it was part of the problem or not. HTH, Or. On Wed, Aug 13, 2014 at 9:22 AM, Or Sher wrote: > Will do the same! > Thanks, > Or. > > > On Tue, Aug 12, 2014 at 6:47 PM, Clint Kelly > wrote: > >> Hi Or, >> >> For now I removed the test that was failing like this from our suite >> and made a note to revisit it in a couple of weeks. Unfortunately I >> still don't know what the issue is. I'll post here if I figure out it >> (please do the same!). My working hypothesis now is that we had some >> kind of OOM problem. >> >> Best regards, >> Clint >> >> On Tue, Aug 12, 2014 at 12:23 AM, Or Sher wrote: >> > Clint, did you find anything? >> > I just noticed it happens to us too on only one node in our CI cluster. >> > I don't think there is a special usage before it happens... The last >> line >> > in the log before the shutdown lines in at least an hour before.. >> > We're using C* 2.0.9. >> > >> > >> > On Thu, Aug 7, 2014 at 12:49 AM, Clint Kelly >> wrote: >> >> >> >> Hi Rob, >> >> >> >> Thanks for the clarification; this is really useful. I'll run some >> >> experiments to see if the problem is a JVM OOM on our build machine. >> >> >> >> Best regards, >> >> Clint >> >> >> >> On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli >> wrote: >> >> > On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli >> >> > wrote: >> >> >> >> >> >> On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands < >> duncan.sa...@gmail.com> >> >> >> wrote: >> >> >>> >> >> >>> this doesn't look like an OOM to me. If the kernel OOM kills >> >> >>> Cassandra >> >> >>> then Cassandra instantly vaporizes, and there will be nothing in >> the >> >> >>> Cassandra logs (you will find information about the OOM in the >> system >> >> >>> logs >> >> >>> though, eg in dmesg). In the log snippet above you see an orderly >> >> >>> shutdown, >> >> >>> this is completely different to the instant OOM kill. >> >> >> >> >> >> >> >> >> Not really. >> >> >> >> >> >> https://issues.apache.org/jira/browse/CASSANDRA-7507 >> >> > >> >> > >> >> > To be clear, there's two different OOMs here, I am talking about the >> JVM >> >> > OOM, not system level. As CASSANDRA-7507 indicates, JVM OOM does not >> >> > necessarily result in the cassandra process dying, and can in fact >> >> > trigger >> >> > clean shutdown. >> >> > >> >> > System level OOM will in fact send the equivalent of KILL, which will >> >> > not >> >> > trigger the clean shutdown hook in Cassandra. >> >> > >> >> > =Rob >> > >> > >> > >> > >> > -- >> > Or Sher >> > > > > -- > Or Sher > -- Or Sher
Re: Cassandra process exiting mysteriously
Will do the same! Thanks, Or. On Tue, Aug 12, 2014 at 6:47 PM, Clint Kelly wrote: > Hi Or, > > For now I removed the test that was failing like this from our suite > and made a note to revisit it in a couple of weeks. Unfortunately I > still don't know what the issue is. I'll post here if I figure out it > (please do the same!). My working hypothesis now is that we had some > kind of OOM problem. > > Best regards, > Clint > > On Tue, Aug 12, 2014 at 12:23 AM, Or Sher wrote: > > Clint, did you find anything? > > I just noticed it happens to us too on only one node in our CI cluster. > > I don't think there is a special usage before it happens... The last > line > > in the log before the shutdown lines in at least an hour before.. > > We're using C* 2.0.9. > > > > > > On Thu, Aug 7, 2014 at 12:49 AM, Clint Kelly > wrote: > >> > >> Hi Rob, > >> > >> Thanks for the clarification; this is really useful. I'll run some > >> experiments to see if the problem is a JVM OOM on our build machine. > >> > >> Best regards, > >> Clint > >> > >> On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli > wrote: > >> > On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli > >> > wrote: > >> >> > >> >> On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands > > >> >> wrote: > >> >>> > >> >>> this doesn't look like an OOM to me. If the kernel OOM kills > >> >>> Cassandra > >> >>> then Cassandra instantly vaporizes, and there will be nothing in the > >> >>> Cassandra logs (you will find information about the OOM in the > system > >> >>> logs > >> >>> though, eg in dmesg). In the log snippet above you see an orderly > >> >>> shutdown, > >> >>> this is completely different to the instant OOM kill. > >> >> > >> >> > >> >> Not really. > >> >> > >> >> https://issues.apache.org/jira/browse/CASSANDRA-7507 > >> > > >> > > >> > To be clear, there's two different OOMs here, I am talking about the > JVM > >> > OOM, not system level. As CASSANDRA-7507 indicates, JVM OOM does not > >> > necessarily result in the cassandra process dying, and can in fact > >> > trigger > >> > clean shutdown. > >> > > >> > System level OOM will in fact send the equivalent of KILL, which will > >> > not > >> > trigger the clean shutdown hook in Cassandra. > >> > > >> > =Rob > > > > > > > > > > -- > > Or Sher > -- Or Sher
Re: Cassandra process exiting mysteriously
Clint, did you find anything? I just noticed it happens to us too on only one node in our CI cluster. I don't think there is a special usage before it happens... The last line in the log before the shutdown lines in at least an hour before.. We're using C* 2.0.9. On Thu, Aug 7, 2014 at 12:49 AM, Clint Kelly wrote: > Hi Rob, > > Thanks for the clarification; this is really useful. I'll run some > experiments to see if the problem is a JVM OOM on our build machine. > > Best regards, > Clint > > On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli wrote: > > On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli > wrote: > >> > >> On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands > >> wrote: > >>> > >>> this doesn't look like an OOM to me. If the kernel OOM kills Cassandra > >>> then Cassandra instantly vaporizes, and there will be nothing in the > >>> Cassandra logs (you will find information about the OOM in the system > logs > >>> though, eg in dmesg). In the log snippet above you see an orderly > shutdown, > >>> this is completely different to the instant OOM kill. > >> > >> > >> Not really. > >> > >> https://issues.apache.org/jira/browse/CASSANDRA-7507 > > > > > > To be clear, there's two different OOMs here, I am talking about the JVM > > OOM, not system level. As CASSANDRA-7507 indicates, JVM OOM does not > > necessarily result in the cassandra process dying, and can in fact > trigger > > clean shutdown. > > > > System level OOM will in fact send the equivalent of KILL, which will not > > trigger the clean shutdown hook in Cassandra. > > > > =Rob > -- Or Sher
Re: Authentication exception
Did you ran a repair after changing replication factor for system_auth ? On Tue, Jul 29, 2014 at 5:48 PM, Jeremy Jongsma wrote: > This is still happening to me; is there anything else I can check? All > nodes have NTP installed, all are in sync, all have open communication to > each other. But usually first thing in the morning, I get this auth > exception. A little while later, it starts working. I'm very puzzled. > > > On Tue, Jul 22, 2014 at 8:53 AM, Jeremy Jongsma > wrote: > >> Verified all clocks are in sync. >> >> >> On Mon, Jul 21, 2014 at 10:03 PM, Rahul Menon wrote: >> >>> I could you perhaps check your ntp? >>> >>> >>> On Tue, Jul 22, 2014 at 3:35 AM, Jeremy Jongsma >>> wrote: >>> >>>> I routinely get this exception from cqlsh on one of my clusters: >>>> >>>> cql.cassandra.ttypes.AuthenticationException: >>>> AuthenticationException(why='org.apache.cassandra.exceptions.ReadTimeoutException: >>>> Operation timed out - received only 2 responses.') >>>> >>>> The system_auth keyspace is set to replicate X times given X nodes in >>>> each datacenter, and at the time of the exception all nodes are reporting >>>> as online and healthy. After a short period (i.e. 30 minutes), it will let >>>> me in again. >>>> >>>> What could be the cause of this? >>>> >>> >>> >> > -- Or Sher
Re: Cassandra coordinator metrics
1. Thanks, make sense. 2. Is there a special reason for that? Is it somewhere in the road map? On Thu, Jul 17, 2014 at 2:45 AM, Tyler Hobbs wrote: > > On Mon, Jul 7, 2014 at 1:08 AM, Or Sher wrote: > >> 1. What exactly does the CoordinatorScanLatency means? >> > > It's the latency for table scans, like "select * from mytable". The > normal read latency only covers reads within a partition. > > >> 2. Is there a write equivalent metric? >> > > No. > > > -- > Tyler Hobbs > DataStax <http://datastax.com/> > -- Or Sher
Cassandra coordinator metrics
Hi, I found that 2.0.9 exposes "CoordinatorReadLatency" and "CoordinatorScanLatency" on a column family resolution. 1. What exactly does the CoordinatorScanLatency means? 2. Is there a write equivalent metric? -- Or Sher
Re: Setting TTL to entire row: UPDATE vs INSERT
That's what I mean, but you stated "you need either to use INSERT or UPDATE
by specifying all the columns in your statement".
How can I use all columns in an update statement? Primary key parts can't
be set..
"This is internal impl details, not supposed to be exposed as public API"
Of course it's an internal implementation, I just thought why wouldn't they
expose an API enhancement like:
update table using ttl 1000 where pk="key value";
If that will only say change the ttl on the row marker, that should be
fairly easy.
On Wed, Jun 11, 2014 at 4:14 PM, DuyHai Doan wrote:
> The only way to set a ttl on an entire existing CQL row is to Insert the
> row again with the same values. Is that correct? -> What do you mean by
> "entire existing CQL3 row" ? Do you mean setting TTL on every column of
> this row ? If so, the answer is yes, you need either to use INSERT or
> UPDATE by specifying all the columns in your statement.
>
> "Wouldn't it be simpler if Cassandra just let us change the ttl on the
> row marker?" --> This is internal impl details, not supposed to be exposed
> as public API
>
>
> On Wed, Jun 11, 2014 at 2:33 PM, Or Sher wrote:
>
>> Thanks DuyHai,
>>
>> Didn't had an idea it works like that is CQL.
>> So per my understanding, the only way to set a ttl on an entire existing
>> CQL row is to Insert the row again with the same values. Is that correct?
>> Wouldn't it be simpler if Cassandra just let us change the ttl on the row
>> marker?
>>
>>
>> On Wed, Jun 11, 2014 at 12:11 PM, DuyHai Doan
>> wrote:
>>
>>> Yes, the TTL is also set on an internal row marker. More details on this
>>> here: https://issues.apache.org/jira/browse/CASSANDRA-6668
>>>
>>>
>>> On Wed, Jun 11, 2014 at 10:38 AM, Or Sher wrote:
>>>
>>>> Hi,
>>>>
>>>> Does that mean internally there is a TTL for an entire CQL row??
>>>> I thought ttl are only attached to CQL row values (Columns which are
>>>> not in the PK).
>>>> I thought when all values of a row are deleted, it should mean that
>>>> that row does not exists.
>>>>
>>>> Please correct me where I'm wrong.
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 11:18 AM, DuyHai Doan
>>>> wrote:
>>>>
>>>>> Hello Or Sher,
>>>>>
>>>>> The behavior is quite normal:
>>>>>
>>>>> 1) insert into test_table (p1,p2,c1,d1,d2) values
>>>>> ('a','b','c','d','e'); --> Insert 5 columns without any TTL
>>>>> 2) update test_table using ttl 10 set d1='---', d2='---' where p1='a'
>>>>> and p2='b' and c1='c'; --> Re-insert columns d1 and d2 with new value and
>>>>> with TTL = 10 secs
>>>>>
>>>>> After a while, colums d1 and d2 expire but not columns p1,p2 and c1
>>>>>
>>>>> 1) insert into test_table (p1,p2,c1,d1,d2) values
>>>>> ('a','b','c','---','---') using ttl 10; --> Insert 5 columns with TTL = 10
>>>>> secs
>>>>>
>>>>> After a while, all 5 columns expire. Cassandra shows
>>>>>
>>>>> (0 rows)
>>>>>
>>>>> because semantically, the row no longer exist (the partition key
>>>>> expired)
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>> Duy Hai DOAN
>>>>>
>>>>>
>>>>> On Wed, Jun 11, 2014 at 8:30 AM, Or Sher wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I encountered a strange phenomena (at least I believe it's strange)
>>>>>> when trying to set a ttl for a whole row.
>>>>>> When trying to set a ttl for a row using update statement and
>>>>>> updating all values I'm getting kind of a "phantom cql row".
>>>>>> When trying to do the same thing using an insert statement it behaves
>>>>>> as expected.
>>>>>>
>>>>>> Here is an example:
>>>>>>
>>>>>> cqlsh:testks> use testks;
>>>>>> cqlsh:testks> create table test_table(
>>>>>> ... p1 text,
>>>>>> ... p2 text,
>>>>>&
Re: Setting TTL to entire row: UPDATE vs INSERT
Thanks DuyHai,
Didn't had an idea it works like that is CQL.
So per my understanding, the only way to set a ttl on an entire existing
CQL row is to Insert the row again with the same values. Is that correct?
Wouldn't it be simpler if Cassandra just let us change the ttl on the row
marker?
On Wed, Jun 11, 2014 at 12:11 PM, DuyHai Doan wrote:
> Yes, the TTL is also set on an internal row marker. More details on this
> here: https://issues.apache.org/jira/browse/CASSANDRA-6668
>
>
> On Wed, Jun 11, 2014 at 10:38 AM, Or Sher wrote:
>
>> Hi,
>>
>> Does that mean internally there is a TTL for an entire CQL row??
>> I thought ttl are only attached to CQL row values (Columns which are not
>> in the PK).
>> I thought when all values of a row are deleted, it should mean that that
>> row does not exists.
>>
>> Please correct me where I'm wrong.
>>
>>
>> On Wed, Jun 11, 2014 at 11:18 AM, DuyHai Doan
>> wrote:
>>
>>> Hello Or Sher,
>>>
>>> The behavior is quite normal:
>>>
>>> 1) insert into test_table (p1,p2,c1,d1,d2) values
>>> ('a','b','c','d','e'); --> Insert 5 columns without any TTL
>>> 2) update test_table using ttl 10 set d1='---', d2='---' where p1='a'
>>> and p2='b' and c1='c'; --> Re-insert columns d1 and d2 with new value and
>>> with TTL = 10 secs
>>>
>>> After a while, colums d1 and d2 expire but not columns p1,p2 and c1
>>>
>>> 1) insert into test_table (p1,p2,c1,d1,d2) values
>>> ('a','b','c','---','---') using ttl 10; --> Insert 5 columns with TTL = 10
>>> secs
>>>
>>> After a while, all 5 columns expire. Cassandra shows
>>>
>>> (0 rows)
>>>
>>> because semantically, the row no longer exist (the partition key
>>> expired)
>>>
>>>
>>> Regards
>>>
>>> Duy Hai DOAN
>>>
>>>
>>> On Wed, Jun 11, 2014 at 8:30 AM, Or Sher wrote:
>>>
>>>> Hi all,
>>>>
>>>> I encountered a strange phenomena (at least I believe it's strange)
>>>> when trying to set a ttl for a whole row.
>>>> When trying to set a ttl for a row using update statement and updating
>>>> all values I'm getting kind of a "phantom cql row".
>>>> When trying to do the same thing using an insert statement it behaves
>>>> as expected.
>>>>
>>>> Here is an example:
>>>>
>>>> cqlsh:testks> use testks;
>>>> cqlsh:testks> create table test_table(
>>>> ... p1 text,
>>>> ... p2 text,
>>>> ... c1 text,
>>>> ... d1 text,
>>>> ... d2 text,
>>>> ... primary key ((p1,p2),c1)
>>>> ... );
>>>> cqlsh:testks> insert into test_table (p1,p2,c1,d1,d2) values
>>>> ('a','b','c','d','e');
>>>> cqlsh:testks> update test_table using ttl 10 set d1='---', d2='---'
>>>> where p1='a' and p2='b' and c1='c';
>>>> cqlsh:testks> select * from test_table;
>>>>
>>>> p1 | p2 | c1 | d1 | d2
>>>> +++-+-
>>>> a | b | c | --- | ---
>>>>
>>>> (1 rows)
>>>>
>>>> cqlsh:testks> select * from test_table;
>>>>
>>>> p1 | p2 | c1 | d1 | d2
>>>> +++--+--
>>>> a | b | c | null | null
>>>>
>>>> (1 rows)
>>>>
>>>> cqlsh:testks> insert into test_table (p1,p2,c1,d1,d2) values
>>>> ('a','b','c','---','---') using ttl 10;
>>>> cqlsh:testks> select * from test_table;
>>>>
>>>> p1 | p2 | c1 | d1 | d2
>>>> +++-+-
>>>> a | b | c | --- | ---
>>>>
>>>> (1 rows)
>>>>
>>>> cqlsh:testks> select * from test_table;
>>>>
>>>> (0 rows)
>>>>
>>>>
>>>>
>>>>
>>>> Is this the expected behavior?
>>>> What's different in the delete and insert statement internally that
>>>> results in such a different behavior?
>>>> We're using C* 2.0.6
>>>>
>>>> Thanks!
>>>> --
>>>> Or Sher
>>>>
>>>
>>>
>>
>>
>> --
>> Or Sher
>>
>
>
--
Or Sher
Re: Setting TTL to entire row: UPDATE vs INSERT
Hi,
Does that mean internally there is a TTL for an entire CQL row??
I thought ttl are only attached to CQL row values (Columns which are not in
the PK).
I thought when all values of a row are deleted, it should mean that that
row does not exists.
Please correct me where I'm wrong.
On Wed, Jun 11, 2014 at 11:18 AM, DuyHai Doan wrote:
> Hello Or Sher,
>
> The behavior is quite normal:
>
> 1) insert into test_table (p1,p2,c1,d1,d2) values ('a','b','c','d','e');
> --> Insert 5 columns without any TTL
> 2) update test_table using ttl 10 set d1='---', d2='---' where p1='a' and
> p2='b' and c1='c'; --> Re-insert columns d1 and d2 with new value and with
> TTL = 10 secs
>
> After a while, colums d1 and d2 expire but not columns p1,p2 and c1
>
> 1) insert into test_table (p1,p2,c1,d1,d2) values
> ('a','b','c','---','---') using ttl 10; --> Insert 5 columns with TTL = 10
> secs
>
> After a while, all 5 columns expire. Cassandra shows
>
> (0 rows)
>
> because semantically, the row no longer exist (the partition key expired)
>
>
> Regards
>
> Duy Hai DOAN
>
>
> On Wed, Jun 11, 2014 at 8:30 AM, Or Sher wrote:
>
>> Hi all,
>>
>> I encountered a strange phenomena (at least I believe it's strange) when
>> trying to set a ttl for a whole row.
>> When trying to set a ttl for a row using update statement and updating
>> all values I'm getting kind of a "phantom cql row".
>> When trying to do the same thing using an insert statement it behaves as
>> expected.
>>
>> Here is an example:
>>
>> cqlsh:testks> use testks;
>> cqlsh:testks> create table test_table(
>> ... p1 text,
>> ... p2 text,
>> ... c1 text,
>> ... d1 text,
>> ... d2 text,
>> ... primary key ((p1,p2),c1)
>> ... );
>> cqlsh:testks> insert into test_table (p1,p2,c1,d1,d2) values
>> ('a','b','c','d','e');
>> cqlsh:testks> update test_table using ttl 10 set d1='---', d2='---' where
>> p1='a' and p2='b' and c1='c';
>> cqlsh:testks> select * from test_table;
>>
>> p1 | p2 | c1 | d1 | d2
>> +++-+-
>> a | b | c | --- | ---
>>
>> (1 rows)
>>
>> cqlsh:testks> select * from test_table;
>>
>> p1 | p2 | c1 | d1 | d2
>> ++----+--+--
>> a | b | c | null | null
>>
>> (1 rows)
>>
>> cqlsh:testks> insert into test_table (p1,p2,c1,d1,d2) values
>> ('a','b','c','---','---') using ttl 10;
>> cqlsh:testks> select * from test_table;
>>
>> p1 | p2 | c1 | d1 | d2
>> +++-+-
>> a | b | c | --- | ---
>>
>> (1 rows)
>>
>> cqlsh:testks> select * from test_table;
>>
>> (0 rows)
>>
>>
>>
>>
>> Is this the expected behavior?
>> What's different in the delete and insert statement internally that
>> results in such a different behavior?
>> We're using C* 2.0.6
>>
>> Thanks!
>> --
>> Or Sher
>>
>
>
--
Or Sher
Setting TTL to entire row: UPDATE vs INSERT
Hi all,
I encountered a strange phenomena (at least I believe it's strange) when
trying to set a ttl for a whole row.
When trying to set a ttl for a row using update statement and updating all
values I'm getting kind of a "phantom cql row".
When trying to do the same thing using an insert statement it behaves as
expected.
Here is an example:
cqlsh:testks> use testks;
cqlsh:testks> create table test_table(
... p1 text,
... p2 text,
... c1 text,
... d1 text,
... d2 text,
... primary key ((p1,p2),c1)
... );
cqlsh:testks> insert into test_table (p1,p2,c1,d1,d2) values
('a','b','c','d','e');
cqlsh:testks> update test_table using ttl 10 set d1='---', d2='---' where
p1='a' and p2='b' and c1='c';
cqlsh:testks> select * from test_table;
p1 | p2 | c1 | d1 | d2
+++-+-
a | b | c | --- | ---
(1 rows)
cqlsh:testks> select * from test_table;
p1 | p2 | c1 | d1 | d2
+++--+--
a | b | c | null | null
(1 rows)
cqlsh:testks> insert into test_table (p1,p2,c1,d1,d2) values
('a','b','c','---','---') using ttl 10;
cqlsh:testks> select * from test_table;
p1 | p2 | c1 | d1 | d2
+++-+-
a | b | c | --- | ---
(1 rows)
cqlsh:testks> select * from test_table;
(0 rows)
Is this the expected behavior?
What's different in the delete and insert statement internally that results
in such a different behavior?
We're using C* 2.0.6
Thanks!
--
Or Sher
Re: Query on Seed node
I'm guessing its just a coincident.. As far as I know, seeds have nothing to do with where the data should be located. I think there could be couple of reasons why you wouldn't see SSTables on a specific column family folder, these are some of them: - You're using a few distinct keys which non of them should be on that seed node. - Node hasn't flushed yet.. You can use nodetool flush to try and flush memtables manually. - You're using manual token assignment and you didn't not assign them well. On Mon, Feb 3, 2014 at 1:25 PM, Aravindan T wrote: > Hi , > > I have a 4 node cassandra cluster with one node marked as seed node. When > i checked the data directory of seed node , it has two folders > /keyspace/columnfamily. > But sstable db files are not available.the folder is empty.The db files > are available in remaining nodes. > > I want to know the reason why db files are not created in seed node ? > what will happen if all the nodes in a cluster is marked as seed node ?? > > > > Aravindan Thangavelu > Tata Consultancy Services > Mailto: aravinda...@tcs.com > Website: http://www.tcs.com > > Experience certainty. IT Services > Business Solutions > Consulting > > > =-=-= > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > -- Or Sher
Re: Upgrading 1.0.9 to 2.0
Thanks. Can I use sstableloader to load SSTables from a RandomPartitioner cluster to a Murmuer3Partitioner cluster? On Thu, Jan 16, 2014 at 9:24 PM, Arya Goudarzi wrote: > Read the upgrade best practices > > http://www.datastax.com/docs/1.1/install/upgrading#best-practices > > You cannot change partitioner > > > http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/architecture/architecturePartitionerAbout_c.html > > > On Thu, Jan 16, 2014 at 2:04 AM, Or Sher wrote: > >> Hi, >> >> In order to upgrade our env from 1.0.9 to 2.0 I thought about the >> following steps: >> >> - Creating a new 1.0.9 cluster >> - Creating the keyspaces and column families >> (I need to move one keyspace data to the new cluster and so:) >> - Moving all xKS SSTables from old cluster to every node in the new >> cluster >> - compact & cleanup >> - upgrading to 1.2.13 (all at once) >> -- upgrade sstables? >> - upgrading to 2.0 (all at once) >> >> 1. I'd like to use new features such as Murmur3 Partitioner and Vnodes - >> How can I accomplish that? >> >> 2. Is there any other features that would be "hard to enable"? >> >> 3. What am I'm missing in the process? >> >> Thanks in advance, >> -- >> Or Sher >> > > > > -- > Cheers, > -Arya > -- Or Sher
Upgrading 1.0.9 to 2.0
Hi, In order to upgrade our env from 1.0.9 to 2.0 I thought about the following steps: - Creating a new 1.0.9 cluster - Creating the keyspaces and column families (I need to move one keyspace data to the new cluster and so:) - Moving all xKS SSTables from old cluster to every node in the new cluster - compact & cleanup - upgrading to 1.2.13 (all at once) -- upgrade sstables? - upgrading to 2.0 (all at once) 1. I'd like to use new features such as Murmur3 Partitioner and Vnodes - How can I accomplish that? 2. Is there any other features that would be "hard to enable"? 3. What am I'm missing in the process? Thanks in advance, -- Or Sher
Re: Moving data from 1.0.9 cluster to a 2.0.* cluster
I think I rather wait until I'll be able to upgrade the current cluster and then make the migration. Thanks! On Thu, Jan 9, 2014 at 8:41 PM, Robert Coli wrote: > On Thu, Jan 9, 2014 at 6:54 AM, Or Sher wrote: > >> I want to use sstableloader in order to load 1.0.9 data to a 2.0.* >> cluster. >> I know that the sstable format is incompatible between the two versions. >> What are my options? >> > > http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra > > If you have a small enough data set or set of target nodes, you could copy > all data from all source nodes to all target nodes and then run cleanup. > > >> Is there a tool to upgrade sstables directly without any real nodes >> involvement? >> > > Offline scrub probably works on older version SSTables and produces latest > version ones? > > > http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/tools/toolsSSTableScrub_t.html > > =Rob > -- Or Sher
Moving data from 1.0.9 cluster to a 2.0.* cluster
Hi all, I want to use sstableloader in order to load 1.0.9 data to a 2.0.* cluster. I know that the sstable format is incompatible between the two versions. What are my options? Is there a tool to upgrade sstables directly without any real nodes involvement? -- Or Sher
Re: nodetool status owns % calculation after upgrade to 2.0.2
RandomPartitioner was the default at < 1.2.* It looks like since 1.2 the default is Murmur3.. Not sure that's your problem if you say you've upgraded from 1.2.*.. On Mon, Jan 6, 2014 at 3:42 AM, Rob Mullen wrote: > Do you know of the default changed? I'm pretty sure I never changed that > setting the the config file. > > Sent from my iPhone > > On Jan 4, 2014, at 11:22 PM, Or Sher wrote: > > Robert, is it possible you've changed the partitioner during the upgrade? > (e.g. from RandomPartitioner to Murmur3Partitioner ?) > > > On Sat, Jan 4, 2014 at 9:32 PM, Mullen, Robert > wrote: > >> The nodetool repair command (which took about 8 hours) seems to have >> sync'd the data in us-east, all 3 nodes returning 59 for the count now. >> I'm wondering if this has more to do with changing the replication factor >> from 2 to 3 and how 2.0.2 reports the % owned rather than the upgrade >> itself. I still don't understand why it's reporting 16% for each node when >> 100% seems to reflect the state of the cluster better. I didn't find any >> info in those issues you posted that would relate to the % changing from >> 100% ->16%. >> >> >> On Sat, Jan 4, 2014 at 12:26 PM, Mullen, Robert < >> robert.mul...@pearson.com> wrote: >> >>> from cql >>> cqlsh>select count(*) from topics; >>> >>> >>> >>> On Sat, Jan 4, 2014 at 12:18 PM, Robert Coli wrote: >>> >>>> On Sat, Jan 4, 2014 at 11:10 AM, Mullen, Robert < >>>> robert.mul...@pearson.com> wrote: >>>> >>>>> I have a column family called "topics" which has a count of 47 on one >>>>> node, 59 on another and 49 on another node. It was my understanding with a >>>>> replication factor of 3 and 3 nodes in each ring that the nodes should be >>>>> equal so I could lose a node in the ring and have no loss of data. Based >>>>> upon that I would expect the counts across the nodes to all be 59 in this >>>>> case. >>>>> >>>> >>>> In what specific way are you counting rows? >>>> >>>> =Rob >>>> >>> >>> >> > > > -- > Or Sher > > -- Or Sher
Re: nodetool status owns % calculation after upgrade to 2.0.2
Robert, is it possible you've changed the partitioner during the upgrade? (e.g. from RandomPartitioner to Murmur3Partitioner ?) On Sat, Jan 4, 2014 at 9:32 PM, Mullen, Robert wrote: > The nodetool repair command (which took about 8 hours) seems to have > sync'd the data in us-east, all 3 nodes returning 59 for the count now. > I'm wondering if this has more to do with changing the replication factor > from 2 to 3 and how 2.0.2 reports the % owned rather than the upgrade > itself. I still don't understand why it's reporting 16% for each node when > 100% seems to reflect the state of the cluster better. I didn't find any > info in those issues you posted that would relate to the % changing from > 100% ->16%. > > > On Sat, Jan 4, 2014 at 12:26 PM, Mullen, Robert > wrote: > >> from cql >> cqlsh>select count(*) from topics; >> >> >> >> On Sat, Jan 4, 2014 at 12:18 PM, Robert Coli wrote: >> >>> On Sat, Jan 4, 2014 at 11:10 AM, Mullen, Robert < >>> robert.mul...@pearson.com> wrote: >>> >>>> I have a column family called "topics" which has a count of 47 on one >>>> node, 59 on another and 49 on another node. It was my understanding with a >>>> replication factor of 3 and 3 nodes in each ring that the nodes should be >>>> equal so I could lose a node in the ring and have no loss of data. Based >>>> upon that I would expect the counts across the nodes to all be 59 in this >>>> case. >>>> >>> >>> In what specific way are you counting rows? >>> >>> =Rob >>> >> >> > -- Or Sher
