Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
> Repair and bootstrap will be limited by the node doing repair or bootstrap, > since it has to do the same amount of work whatever num_tokens is. That's what I was thinking. I normally assume repair has very little data to stream, and most of the time is taken creating the merkle trees. In that case the node repair was started on still has to compact / hash all it's data, however hashing the replicas would go faster is it's done on more machines. Cheers - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 13/08/2013, at 11:13 PM, Richard Low wrote: > On 13 August 2013 10:15, Alain RODRIGUEZ wrote: > > Streaming from all the physical nodes in the cluster should make repair > faster, for the same reason it makes bootstrap faster. Shouldn't it ? > > Virtual nodes doesn't speed up either very much. Repair and bootstrap will > be limited by the node doing repair or bootstrap, since it has to do the same > amount of work whatever num_tokens is. It places a more even load across the > rest of the cluster though, since it will repair with or bootstrap from all > nodes in the cluster. So the overall time will in most cases be about the > same. > > The real speedup from vnodes comes when running removenode, when the > streaming happens in parallel across all nodes. > > Richard.
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
On 13 August 2013 10:15, Alain RODRIGUEZ wrote: Streaming from all the physical nodes in the cluster should make repair > faster, for the same reason it makes bootstrap faster. Shouldn't it ? > Virtual nodes doesn't speed up either very much. Repair and bootstrap will be limited by the node doing repair or bootstrap, since it has to do the same amount of work whatever num_tokens is. It places a more even load across the rest of the cluster though, since it will repair with or bootstrap from all nodes in the cluster. So the overall time will in most cases be about the same. The real speedup from vnodes comes when running removenode, when the streaming happens in parallel across all nodes. Richard.
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
@Aaron Streaming from all the physical nodes in the cluster should make repair faster, for the same reason it makes bootstrap faster. Shouldn't it ? 2013/8/12 Aaron Morton > Aaron - I read about the virtual nodes at > http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 > > > Thanks, I did not see anything in there about making repair smoother / > faster. > > Cheers > A > - > Aaron Morton > Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 10/08/2013, at 11:39 AM, rash aroskar wrote: > > Aaron - I read about the virtual nodes at > http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 > > > > > > > > > On Tue, Aug 6, 2013 at 4:49 AM, Richard Low wrote: > >> On 6 August 2013 08:40, Aaron Morton wrote: >> >>> The reason for me looking at virtual nodes is because of terrible >>> experiences we had with 0.8 repairs and as per documentation (an logically) >>> the virtual nodes seems like it will help repairs being smoother. Is this >>> true? >>> >>> I've not thought too much about how they help repair run smoother, what >>> was the documentation you read ? >>> >> >> There might be a slight improvement but I haven't observed any. The >> difference might be that, because every node shares replicas with every >> other (with high probability), a single repair operation does the same work >> on the node it was called on, but the rest is spread out over the cluster, >> rather than just the RF nodes either side of the repairing node. This >> means the post-repair compaction work will take less time and the length of >> time a node is loaded for during repair is less. >> >> However, the other benefits of vnodes are likely to be much more useful. >> >> Richard. >> > > >
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
> Aaron - I read about the virtual nodes at > http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 Thanks, I did not see anything in there about making repair smoother / faster. Cheers A - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 10/08/2013, at 11:39 AM, rash aroskar wrote: > Aaron - I read about the virtual nodes at > http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 > > > > > > > > > On Tue, Aug 6, 2013 at 4:49 AM, Richard Low wrote: > On 6 August 2013 08:40, Aaron Morton wrote: >> The reason for me looking at virtual nodes is because of terrible >> experiences we had with 0.8 repairs and as per documentation (an logically) >> the virtual nodes seems like it will help repairs being smoother. Is this >> true? > I've not thought too much about how they help repair run smoother, what was > the documentation you read ? > > There might be a slight improvement but I haven't observed any. The > difference might be that, because every node shares replicas with every other > (with high probability), a single repair operation does the same work on the > node it was called on, but the rest is spread out over the cluster, rather > than just the RF nodes either side of the repairing node. This means the > post-repair compaction work will take less time and the length of time a node > is loaded for during repair is less. > > However, the other benefits of vnodes are likely to be much more useful. > > Richard. >
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Aaron - I read about the virtual nodes at http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 On Tue, Aug 6, 2013 at 4:49 AM, Richard Low wrote: > On 6 August 2013 08:40, Aaron Morton wrote: > >> The reason for me looking at virtual nodes is because of terrible >> experiences we had with 0.8 repairs and as per documentation (an logically) >> the virtual nodes seems like it will help repairs being smoother. Is this >> true? >> >> I've not thought too much about how they help repair run smoother, what >> was the documentation you read ? >> > > There might be a slight improvement but I haven't observed any. The > difference might be that, because every node shares replicas with every > other (with high probability), a single repair operation does the same work > on the node it was called on, but the rest is spread out over the cluster, > rather than just the RF nodes either side of the repairing node. This > means the post-repair compaction work will take less time and the length of > time a node is loaded for during repair is less. > > However, the other benefits of vnodes are likely to be much more useful. > > Richard. >
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
On 6 August 2013 08:40, Aaron Morton wrote: > The reason for me looking at virtual nodes is because of terrible > experiences we had with 0.8 repairs and as per documentation (an logically) > the virtual nodes seems like it will help repairs being smoother. Is this > true? > > I've not thought too much about how they help repair run smoother, what > was the documentation you read ? > There might be a slight improvement but I haven't observed any. The difference might be that, because every node shares replicas with every other (with high probability), a single repair operation does the same work on the node it was called on, but the rest is spread out over the cluster, rather than just the RF nodes either side of the repairing node. This means the post-repair compaction work will take less time and the length of time a node is loaded for during repair is less. However, the other benefits of vnodes are likely to be much more useful. Richard.
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
> The reason for me looking at virtual nodes is because of terrible experiences > we had with 0.8 repairs and as per documentation (an logically) the virtual > nodes seems like it will help repairs being smoother. Is this true? I've not thought too much about how they help repair run smoother, what was the documentation you read ? > Also how to get the right number of virtual nodes? Use the default 256 Hope that helps. - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 3/08/2013, at 7:39 AM, rash aroskar wrote: > Thanks for helpful responses. The upgrade from 0.8 to 1.2 is not direct, we > have setup test cluster where we did upgrade from 0.8 to 1.1 and then 1.2. > Also we will do a whole different cluster with 1.2, the 0.8 cluster will not > be upgraded. But the data will be moved from 0.8 cluster to 1.2 cluster. > The reason for me looking at virtual nodes is because of terrible experiences > we had with 0.8 repairs and as per documentation (an logically) the virtual > nodes seems like it will help repairs being smoother. Is this true? Also how > to get the right number of virtual nodes? David suggested 64 vnodes for 20 > machines. Is there a formula or a thought process to be followed to get this > number right? > > > On Mon, Jul 29, 2013 at 4:15 AM, aaron morton wrote: > I would *strongly* recommend against upgrading from 0.8 directly to 1.2. > Skipping a major version is generally not recommended, skipped 3 would seem > like carelessness. > >> I second Romain, do the upgrade and make sure the health is good first. > > +1 but I would also recommend deciding if you actually need to use virtual > nodes. The shuffle process can take a long time and people have had mixed > experiences with it. > > If you wanted to move to 1.2 and get vNodes I would consider spinning up a > new cluster and bulk loading into it. You could do an initial load and then > to delta loads using snapshots, there would however be a period of stale data > in the new cluster until the last delta snapshot is loaded. > > Cheers > > - > Aaron Morton > Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 27/07/2013, at 3:36 AM, David McNelis wrote: > >> I second Romain, do the upgrade and make sure the health is good first. >> >> If you have or plan to have a large number of nodes, you might consider >> using fewer than 256 as your initial vnodes amount. I think that number is >> inflated from reasonable in the docs, as we've had some people talk about >> potential performance degradation if you have a large number of nodes and a >> very high number of vnodes, if I had it to do over again, I'd have done 64 >> vnodes as my default (across 20 nodes). >> >> Another thing to be very cognizant of before shuffle is disk space. You >> *must* have less than 50% used in order to do the shuffle successfully >> because no data is removed (cleaned) from a node during the shuffle process >> and the shuffle process essentially doubles the amount of data until you're >> able to run a clean. >> >> >> On Fri, Jul 26, 2013 at 11:25 AM, Romain HARDOUIN >> wrote: >> Vnodes are a great feature. More nodes are involved during operations such >> as bootstrap, decommission, etc. >> DataStax documentation is definitely a must read. >> That said, If I were you, I'd wait somewhat before to shuffle the ring. I'd >> focus on cluster upgrade and monitoring the nodes. (number of files handles, >> memory usage, latency, etc). >> Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since >> then. Be careful about compaction strategies you choose and double check the >> options. >> >> Regards, >> Romain >> >> rash aroskar a écrit sur 25/07/2013 23:25:11 : >> >> > De : rash aroskar >> > A : user@cassandra.apache.org, >> > Date : 25/07/2013 23:25 >> > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons? >> > >> > Hi, >> > I am upgrading my cassandra cluster from 0.8 to 1.2.5. >> > In cassandra 1.2.5 the 'num_token' attribute confuses me. >> > I understand that it distributes multiple tokens per node but I am >> > not clear how that is helpful for performance or load balancing. Can >> > anyone elaborate? has anyone used this feature and knows its >> > advantages/disadvantages? >> > >> > Thanks, >> > Rashmi >> > >
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Thanks for helpful responses. The upgrade from 0.8 to 1.2 is not direct, we have setup test cluster where we did upgrade from 0.8 to 1.1 and then 1.2. Also we will do a whole different cluster with 1.2, the 0.8 cluster will not be upgraded. But the data will be moved from 0.8 cluster to 1.2 cluster. The reason for me looking at virtual nodes is because of terrible experiences we had with 0.8 repairs and as per documentation (an logically) the virtual nodes seems like it will help repairs being smoother. Is this true? Also how to get the right number of virtual nodes? David suggested 64 vnodes for 20 machines. Is there a formula or a thought process to be followed to get this number right? On Mon, Jul 29, 2013 at 4:15 AM, aaron morton wrote: > I would *strongly* recommend against upgrading from 0.8 directly to 1.2. > Skipping a major version is generally not recommended, skipped 3 would seem > like carelessness. > > I second Romain, do the upgrade and make sure the health is good first. > > +1 but I would also recommend deciding if you actually need to use virtual > nodes. The shuffle process can take a long time and people have had mixed > experiences with it. > > If you wanted to move to 1.2 and get vNodes I would consider spinning up a > new cluster and bulk loading into it. You could do an initial load and then > to delta loads using snapshots, there would however be a period of stale > data in the new cluster until the last delta snapshot is loaded. > > Cheers > > - > Aaron Morton > Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 27/07/2013, at 3:36 AM, David McNelis wrote: > > I second Romain, do the upgrade and make sure the health is good first. > > If you have or plan to have a large number of nodes, you might consider > using fewer than 256 as your initial vnodes amount. I think that number is > inflated from reasonable in the docs, as we've had some people talk about > potential performance degradation if you have a large number of nodes and a > very high number of vnodes, if I had it to do over again, I'd have done 64 > vnodes as my default (across 20 nodes). > > Another thing to be very cognizant of before shuffle is disk space. You > *must* have less than 50% used in order to do the shuffle successfully > because no data is removed (cleaned) from a node during the shuffle process > and the shuffle process essentially doubles the amount of data until you're > able to run a clean. > > > On Fri, Jul 26, 2013 at 11:25 AM, Romain HARDOUIN < > romain.hardo...@urssaf.fr> wrote: > >> Vnodes are a great feature. More nodes are involved during operations >> such as bootstrap, decommission, etc. >> DataStax documentation is definitely a must read. >> That said, If I were you, I'd wait somewhat before to shuffle the ring. >> I'd focus on cluster upgrade and monitoring the nodes. (number of files >> handles, memory usage, latency, etc). >> Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since >> then. Be careful about compaction strategies you choose and double check >> the options. >> >> Regards, >> Romain >> >> rash aroskar a écrit sur 25/07/2013 23:25:11 : >> >> > De : rash aroskar >> > A : user@cassandra.apache.org, >> > Date : 25/07/2013 23:25 >> > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons? >> > >> > Hi, >> > I am upgrading my cassandra cluster from 0.8 to 1.2.5. >> > In cassandra 1.2.5 the 'num_token' attribute confuses me. >> > I understand that it distributes multiple tokens per node but I am >> > not clear how that is helpful for performance or load balancing. Can >> > anyone elaborate? has anyone used this feature and knows its >> > advantages/disadvantages? >> > >> > Thanks, >> > Rashmi >> > > >
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
I would *strongly* recommend against upgrading from 0.8 directly to 1.2. Skipping a major version is generally not recommended, skipped 3 would seem like carelessness. > I second Romain, do the upgrade and make sure the health is good first. +1 but I would also recommend deciding if you actually need to use virtual nodes. The shuffle process can take a long time and people have had mixed experiences with it. If you wanted to move to 1.2 and get vNodes I would consider spinning up a new cluster and bulk loading into it. You could do an initial load and then to delta loads using snapshots, there would however be a period of stale data in the new cluster until the last delta snapshot is loaded. Cheers - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 27/07/2013, at 3:36 AM, David McNelis wrote: > I second Romain, do the upgrade and make sure the health is good first. > > If you have or plan to have a large number of nodes, you might consider using > fewer than 256 as your initial vnodes amount. I think that number is > inflated from reasonable in the docs, as we've had some people talk about > potential performance degradation if you have a large number of nodes and a > very high number of vnodes, if I had it to do over again, I'd have done 64 > vnodes as my default (across 20 nodes). > > Another thing to be very cognizant of before shuffle is disk space. You > *must* have less than 50% used in order to do the shuffle successfully > because no data is removed (cleaned) from a node during the shuffle process > and the shuffle process essentially doubles the amount of data until you're > able to run a clean. > > > On Fri, Jul 26, 2013 at 11:25 AM, Romain HARDOUIN > wrote: > Vnodes are a great feature. More nodes are involved during operations such as > bootstrap, decommission, etc. > DataStax documentation is definitely a must read. > That said, If I were you, I'd wait somewhat before to shuffle the ring. I'd > focus on cluster upgrade and monitoring the nodes. (number of files handles, > memory usage, latency, etc). > Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since > then. Be careful about compaction strategies you choose and double check the > options. > > Regards, > Romain > > rash aroskar a écrit sur 25/07/2013 23:25:11 : > > > De : rash aroskar > > A : user@cassandra.apache.org, > > Date : 25/07/2013 23:25 > > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons? > > > > Hi, > > I am upgrading my cassandra cluster from 0.8 to 1.2.5. > > In cassandra 1.2.5 the 'num_token' attribute confuses me. > > I understand that it distributes multiple tokens per node but I am > > not clear how that is helpful for performance or load balancing. Can > > anyone elaborate? has anyone used this feature and knows its > > advantages/disadvantages? > > > > Thanks, > > Rashmi >
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
I second Romain, do the upgrade and make sure the health is good first. If you have or plan to have a large number of nodes, you might consider using fewer than 256 as your initial vnodes amount. I think that number is inflated from reasonable in the docs, as we've had some people talk about potential performance degradation if you have a large number of nodes and a very high number of vnodes, if I had it to do over again, I'd have done 64 vnodes as my default (across 20 nodes). Another thing to be very cognizant of before shuffle is disk space. You *must* have less than 50% used in order to do the shuffle successfully because no data is removed (cleaned) from a node during the shuffle process and the shuffle process essentially doubles the amount of data until you're able to run a clean. On Fri, Jul 26, 2013 at 11:25 AM, Romain HARDOUIN wrote: > Vnodes are a great feature. More nodes are involved during operations such > as bootstrap, decommission, etc. > DataStax documentation is definitely a must read. > That said, If I were you, I'd wait somewhat before to shuffle the ring. > I'd focus on cluster upgrade and monitoring the nodes. (number of files > handles, memory usage, latency, etc). > Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since > then. Be careful about compaction strategies you choose and double check > the options. > > Regards, > Romain > > rash aroskar a écrit sur 25/07/2013 23:25:11 : > > > De : rash aroskar > > A : user@cassandra.apache.org, > > Date : 25/07/2013 23:25 > > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons? > > > > Hi, > > I am upgrading my cassandra cluster from 0.8 to 1.2.5. > > In cassandra 1.2.5 the 'num_token' attribute confuses me. > > I understand that it distributes multiple tokens per node but I am > > not clear how that is helpful for performance or load balancing. Can > > anyone elaborate? has anyone used this feature and knows its > > advantages/disadvantages? > > > > Thanks, > > Rashmi >
RE: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Vnodes are a great feature. More nodes are involved during operations such as bootstrap, decommission, etc. DataStax documentation is definitely a must read. That said, If I were you, I'd wait somewhat before to shuffle the ring. I'd focus on cluster upgrade and monitoring the nodes. (number of files handles, memory usage, latency, etc). Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since then. Be careful about compaction strategies you choose and double check the options. Regards, Romain rash aroskar a écrit sur 25/07/2013 23:25:11 : > De : rash aroskar > A : user@cassandra.apache.org, > Date : 25/07/2013 23:25 > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons? > > Hi, > I am upgrading my cassandra cluster from 0.8 to 1.2.5. > In cassandra 1.2.5 the 'num_token' attribute confuses me. > I understand that it distributes multiple tokens per node but I am > not clear how that is helpful for performance or load balancing. Can > anyone elaborate? has anyone used this feature and knows its > advantages/disadvantages? > > Thanks, > Rashmi
RE: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Disadvantage, hadoop support is based on the token range, so maps may equals numtokens per node * nodes number. Best Regards,Andy Hu（胡 珊） Date: Thu, 25 Jul 2013 17:25:11 -0400 Subject: cassandra 1.2.5- virtual nodes (num_token) pros/cons? From: rashmi.aros...@gmail.com To: user@cassandra.apache.org Hi,I am upgrading my cassandra cluster from 0.8 to 1.2.5. In cassandra 1.2.5 the 'num_token' attribute confuses me. I understand that it distributes multiple tokens per node but I am not clear how that is helpful for performance or load balancing. Can anyone elaborate? has anyone used this feature and knows its advantages/disadvantages? Thanks, Rashmi
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
It is very useful to upgrade the apps perfomance. For example, if you have a machine with X capacity, you can put the num_token=256. If you add a machine in your cluster with (X*2) capacity you can put the num_token=512. So, this new machine will receive twice the load automatically. Moreover, you can minimize how long a machine replacement is going to take and will minimize the time to repair a disk failure. Read the docs to more informations: http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 Cheers! 2013/7/25 rash aroskar > Hi, > I am upgrading my cassandra cluster from 0.8 to 1.2.5. > In cassandra 1.2.5 the 'num_token' attribute confuses me. > I understand that it distributes multiple tokens per node but I am not > clear how that is helpful for performance or load balancing. Can anyone > elaborate? has anyone used this feature and knows its > advantages/disadvantages? > > Thanks, > Rashmi > -- Atenciosamente, Sávio S. Teles de Oliveira voice: +55 62 9136 6996 http://br.linkedin.com/in/savioteles Mestrando em Ciências da Computação - UFG Arquiteto de Software Laboratory for Ubiquitous and Pervasive Applications (LUPA) - UFG
cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Hi, I am upgrading my cassandra cluster from 0.8 to 1.2.5. In cassandra 1.2.5 the 'num_token' attribute confuses me. I understand that it distributes multiple tokens per node but I am not clear how that is helpful for performance or load balancing. Can anyone elaborate? has anyone used this feature and knows its advantages/disadvantages? Thanks, Rashmi
