[ https://issues.apache.org/jira/browse/CASSANDRA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2948: -------------------------------------- Reviewer: brandon.williams Assignee: Sam Overton (was: Brandon Williams) > Nodetool move fails to stream out data from moved node to new endpoint. > ----------------------------------------------------------------------- > > Key: CASSANDRA-2948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2948 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.8.2 > Reporter: Sam Overton > Assignee: Sam Overton > Priority: Critical > Labels: move, streaming > Fix For: 0.8.3 > > Attachments: 2948.patch > > > When moving a node in the ring with nodetool move, that node streams its data > to itself instead of to the new endpoint responsible for its old range. > Steps to reproduce: > * Create a cluster (A,B,C,D) with tokens (0,4,8,C) using > ByteOrderedPartitioner > * Create a keyspace and CF with RF=1 > * Insert keys (2,6,A,E). This should put one key on each node. > * Move node A to token 7. This should cause: > + node C streams key 6 to node A > + node A streams key E to node B > However instead, node A streams key E to itself. > Selected log messages from node A: > INFO [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:17,075 > StorageService.java (line 1878) Moving miles/10.2.129.41 from > Token(bytes[00]) to Token(bytes[07]). > DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:17,080 > StorageService.java (line 1941) Table ks: work map > {/10.2.129.16=[(Token(bytes[04]),Token(bytes[07])]]}. > INFO [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:17,080 > StorageService.java (line 1946) Sleeping 30000 ms before start > streaming/fetching ranges. > ... > INFO [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,728 > StorageService.java (line 522) Moving: fetching new ranges and streaming old > ranges > DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,728 > StorageService.java (line 1960) [Move->STREAMING] Work Map: > {ks={(Token(bytes[0c]),Token(bytes[00])]=[miles/10.2.129.41]}} > DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,729 > StorageService.java (line 1965) [Move->FETCHING] Work Map: > {ks={/10.2.129.16=[(Token(bytes[04]),Token(bytes[07])]]}} > DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,730 > StorageService.java (line 2411) Requesting from /10.2.129.16 ranges > (Token(bytes[04]),Token(bytes[07])] > ... > INFO [StreamStage:1] 2011-07-26 16:29:46,737 StreamOut.java (line 90) > Beginning transfer to miles/10.2.129.41 > DEBUG [StreamStage:1] 2011-07-26 16:29:46,737 StreamOut.java (line 91) Ranges > are (Token(bytes[0c]),Token(bytes[00])] > This appears to be caused because in StorageService.move we call > Gossiper.instance.addLocalApplicationState(ApplicationState.STATUS, > valueFactory.moving(newToken)); > and then get the new token metadata in order to calculate where the new > endpoint is that we should stream to > TokenMetadata tokenMetaClone = tokenMetadata_.cloneAfterAllSettled(); > however, in addLocalApplicationState there is no notification broadcast for > the change in local state, so tokenMetadata_ never updates the list of moving > nodes, and the tokenMetaClone is still the state of the ring from before the > move. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira