[jira] [Commented] (CASSANDRA-4658) NTS/RackAwareness doesn't work with vnodes
[ https://issues.apache.org/jira/browse/CASSANDRA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455016#comment-13455016 ] Brandon Williams commented on CASSANDRA-4658: - What if we treated each rack like its own ring, similar to DCs? If a key naturally hashed to node A in rack1, then it would hash closest to node B *within* rack2. This means you'd need an approximately balanced ring inside each rack, but vnodes should actually help in this instance and largely solve that. The only missing piece would be a consistent ordering of racks themselves (r1-r2-r3) but that should be a bit easier to do. NTS/RackAwareness doesn't work with vnodes -- Key: CASSANDRA-4658 URL: https://issues.apache.org/jira/browse/CASSANDRA-4658 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0 beta 1 Reporter: Nick Bailey Fix For: 1.2.0 It doesn't look like the current vnode placement strategy will work with people using NTS and multiple racks. For reasons also described on CASSANDRA-3810, using racks and NTS requires tokens to alternate racks around the ring in order to get an even distribution of data. The current solution for upgrading/placing vnodes won't take this into account and will likely generate some hotspots around the ring. Not sure what the best solution is. The two immediately obvious approaches appear to be quite complicated at first. * Fixing NTS to remove the requirement for rack ordering ** No idea how this would be accomplished ** Presents challenges for people upgrading. Would need to deprecate NTS for a new strategy that replaces it, then have a clear upgrade path to that strategy which would need to be in a pre 1.2 release. * Changing vnode placement strategy ** Ordering vnodes would require quite a bit of additional logic. Adding a new node or rack would also need to maintain ordering which could cause enough data movement to remove any benefits vnodes have already. ** We could potentially adjust vnode token placement to offset any imbalances caused by NTS but this would require a fairly intelligent mechanism for determining vnode placement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4658) NTS/RackAwareness doesn't work with vnodes
[ https://issues.apache.org/jira/browse/CASSANDRA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455064#comment-13455064 ] Vijay commented on CASSANDRA-4658: -- I think, the current NTS works when we manually specify the tokens to be in different RAC's, that is true in vnode's case too if we manually assign tokens... To automate we can do more in getRandomTokens() so we check if the random tokens are in different racks (I Understand that the probability of the random token changes by doing this)... Agree? Above will only solve the case of a newly bootstrapped node... for the rest just replace the node? NTS/RackAwareness doesn't work with vnodes -- Key: CASSANDRA-4658 URL: https://issues.apache.org/jira/browse/CASSANDRA-4658 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0 beta 1 Reporter: Nick Bailey Fix For: 1.2.0 It doesn't look like the current vnode placement strategy will work with people using NTS and multiple racks. For reasons also described on CASSANDRA-3810, using racks and NTS requires tokens to alternate racks around the ring in order to get an even distribution of data. The current solution for upgrading/placing vnodes won't take this into account and will likely generate some hotspots around the ring. Not sure what the best solution is. The two immediately obvious approaches appear to be quite complicated at first. * Fixing NTS to remove the requirement for rack ordering ** No idea how this would be accomplished ** Presents challenges for people upgrading. Would need to deprecate NTS for a new strategy that replaces it, then have a clear upgrade path to that strategy which would need to be in a pre 1.2 release. * Changing vnode placement strategy ** Ordering vnodes would require quite a bit of additional logic. Adding a new node or rack would also need to maintain ordering which could cause enough data movement to remove any benefits vnodes have already. ** We could potentially adjust vnode token placement to offset any imbalances caused by NTS but this would require a fairly intelligent mechanism for determining vnode placement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4658) NTS/RackAwareness doesn't work with vnodes
[ https://issues.apache.org/jira/browse/CASSANDRA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455075#comment-13455075 ] Brandon Williams commented on CASSANDRA-4658: - bq. To automate we can do more in getRandomTokens() so we check if the random tokens are in different racks That's impossible to do in a new cluster, though. NTS/RackAwareness doesn't work with vnodes -- Key: CASSANDRA-4658 URL: https://issues.apache.org/jira/browse/CASSANDRA-4658 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0 beta 1 Reporter: Nick Bailey Fix For: 1.2.0 It doesn't look like the current vnode placement strategy will work with people using NTS and multiple racks. For reasons also described on CASSANDRA-3810, using racks and NTS requires tokens to alternate racks around the ring in order to get an even distribution of data. The current solution for upgrading/placing vnodes won't take this into account and will likely generate some hotspots around the ring. Not sure what the best solution is. The two immediately obvious approaches appear to be quite complicated at first. * Fixing NTS to remove the requirement for rack ordering ** No idea how this would be accomplished ** Presents challenges for people upgrading. Would need to deprecate NTS for a new strategy that replaces it, then have a clear upgrade path to that strategy which would need to be in a pre 1.2 release. * Changing vnode placement strategy ** Ordering vnodes would require quite a bit of additional logic. Adding a new node or rack would also need to maintain ordering which could cause enough data movement to remove any benefits vnodes have already. ** We could potentially adjust vnode token placement to offset any imbalances caused by NTS but this would require a fairly intelligent mechanism for determining vnode placement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4658) NTS/RackAwareness doesn't work with vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455080#comment-13455080
]
Vijay commented on CASSANDRA-4658:
--
{quote}
That's impossible to do in a new cluster, though.
{quote}
What i was thinking was: When the First node is added to Rack1 then we dont
know about RAC2 and 3, so we use a purely go with random tokens When RAC2
node comes online it should compare and take spots which will intersecting the
other RAC1. When RAC3 is added then they have to intersect RAC1 and 2. I am not
saying this is perfect solution we can at-least reduce the probability of the
ranges being in the same RAC. (We can complicate the logic as much as we want
to get a even proportion during bootstrap phase... :) ).
NTS/RackAwareness doesn't work with vnodes
--
Key: CASSANDRA-4658
URL: https://issues.apache.org/jira/browse/CASSANDRA-4658
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.0 beta 1
Reporter: Nick Bailey
Fix For: 1.2.0
It doesn't look like the current vnode placement strategy will work with
people using NTS and multiple racks.
For reasons also described on CASSANDRA-3810, using racks and NTS requires
tokens to alternate racks around the ring in order to get an even
distribution of data. The current solution for upgrading/placing vnodes won't
take this into account and will likely generate some hotspots around the
ring.
Not sure what the best solution is. The two immediately obvious approaches
appear to be quite complicated at first.
* Fixing NTS to remove the requirement for rack ordering
** No idea how this would be accomplished
** Presents challenges for people upgrading. Would need to deprecate NTS for
a new strategy that replaces it, then have a clear upgrade path to that
strategy which would need to be in a pre 1.2 release.
* Changing vnode placement strategy
** Ordering vnodes would require quite a bit of additional logic. Adding a
new node or rack would also need to maintain ordering which could cause
enough data movement to remove any benefits vnodes have already.
** We could potentially adjust vnode token placement to offset any imbalances
caused by NTS but this would require a fairly intelligent mechanism for
determining vnode placement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4658) NTS/RackAwareness doesn't work with vnodes
[ https://issues.apache.org/jira/browse/CASSANDRA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455222#comment-13455222 ] Nick Bailey commented on CASSANDRA-4658: So Brandon's solution isn't really much different from what NTS already does. After thinking about it some more though, it could be that the random placement of tokens will naturally resolve any imbalances, assuming you have a largish number of tokens. Brandon pointed out offline, that the random placement of tokens will at least approach alternating racks around the ring. We should probably do some testing to see if this is really an issue. NTS/RackAwareness doesn't work with vnodes -- Key: CASSANDRA-4658 URL: https://issues.apache.org/jira/browse/CASSANDRA-4658 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0 beta 1 Reporter: Nick Bailey Fix For: 1.2.0 It doesn't look like the current vnode placement strategy will work with people using NTS and multiple racks. For reasons also described on CASSANDRA-3810, using racks and NTS requires tokens to alternate racks around the ring in order to get an even distribution of data. The current solution for upgrading/placing vnodes won't take this into account and will likely generate some hotspots around the ring. Not sure what the best solution is. The two immediately obvious approaches appear to be quite complicated at first. * Fixing NTS to remove the requirement for rack ordering ** No idea how this would be accomplished ** Presents challenges for people upgrading. Would need to deprecate NTS for a new strategy that replaces it, then have a clear upgrade path to that strategy which would need to be in a pre 1.2 release. * Changing vnode placement strategy ** Ordering vnodes would require quite a bit of additional logic. Adding a new node or rack would also need to maintain ordering which could cause enough data movement to remove any benefits vnodes have already. ** We could potentially adjust vnode token placement to offset any imbalances caused by NTS but this would require a fairly intelligent mechanism for determining vnode placement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4658) NTS/RackAwareness doesn't work with vnodes
[ https://issues.apache.org/jira/browse/CASSANDRA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13454159#comment-13454159 ] Nick Bailey commented on CASSANDRA-4658: We could potentially say that rack awareness and vnodes are incompatible. Which will essentially mean that cassandra can't guarantee that losing a single rack won't bring down an entire replica set. That makes the problem slightly easier, but still requires some sort of clear upgrade path for people on NTS with racks. Changing the replication strategy or changing the snitch to report different racks won't cause data to be streamed to any new owners. So we would need to come up with a mechanism for updating without data loss. NTS/RackAwareness doesn't work with vnodes -- Key: CASSANDRA-4658 URL: https://issues.apache.org/jira/browse/CASSANDRA-4658 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0 beta 1 Reporter: Nick Bailey Fix For: 1.2.0 It doesn't look like the current vnode placement strategy will work with people using NTS and multiple racks. For reasons also described on CASSANDRA-3810, using racks and NTS requires tokens to alternate racks around the ring in order to get an even distribution of data. The current solution for upgrading/placing vnodes won't take this into account and will likely generate some hotspots around the ring. Not sure what the best solution is. The two immediately obvious approaches appear to be quite complicated at first. * Fixing NTS to remove the requirement for rack ordering ** No idea how this would be accomplished ** Presents challenges for people upgrading. Would need to deprecate NTS for a new strategy that replaces it, then have a clear upgrade path to that strategy which would need to be in a pre 1.2 release. * Changing vnode placement strategy ** Ordering vnodes would require quite a bit of additional logic. Adding a new node or rack would also need to maintain ordering which could cause enough data movement to remove any benefits vnodes have already. ** We could potentially adjust vnode token placement to offset any imbalances caused by NTS but this would require a fairly intelligent mechanism for determining vnode placement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
