Thanks Jim, looks I'll have to read into the code to understand what is happening under the hood
yang On Mon, Sep 10, 2012 at 9:45 AM, Jim Cistaro <jcist...@netflix.com> wrote: > We have seen various issues from these replaced nodes hanging around. > For clusters where a lot of nodes have been replaced, we see these > replaced nodes having an impact on heap/GC and a lot of tcp > timeouts/retransmits (because the old nodes no longer exist). As a result, > we have begun cleaning these up using unsafeAssassinateEndpoint via jmx. > We have only started using recently. So far no bad side effects. This > also helps because those replaced nodes can appear as "unreachable nodes" > wrt schema and sometimes prevent things like CF truncation. > > Using unsafeAssassinateEndpoint will clean these from unreachable nodes > and will mark them as LEFT in gossip info. There is a ttl for them in > gossipinfo and they should go away after 3 days. Once they are marked > LEFT, you should stop seeing those up/same/dead messages. > > unsafeAssassinateEndpoint is "unsafe" in that, if you specify IP of a > real node in cluster, that node will be assassinated. Otherwise, if you > specify nodes that have been replaced, it is supposed to work correctly. > > Hope this helps, > jc > > > From: Yang <teddyyyy...@gmail.com> > Reply-To: <user@cassandra.apache.org> > Date: Mon, 10 Sep 2012 01:10:56 -0700 > To: <user@cassandra.apache.org> > Subject: replace_token code? > > it looks that by specifying replace_token, the old owner is not removed > from gossip (which I had thought it would do). > Then it's understandable that the old owner would resurface later and we > get some warning saying that the same token is owned by both. > > > I ran an example with a 2-node cluster, with RF=2. host 10.72.201.80 > was running for a while and had some data, then i shut it down, and > booted up 10.190.221.204 with replace_token of the old token owned by the > previous host. > the following log sequence shows that the new host does acquire the token, > but it does not at the same time remove 80 forcefully from gossip. > instead, a few seconds later, it believed that .80 became live again. > I don't have much understanding of the Gossip protocol, but roughly know > that it's probability-based, looks we need an "assertive"/"NOW" > membership control message for replace_token. > > > > > thanks > yang > > > WARN [main] 2012-09-10 08:00:21,855 TokenMetadata.java (line 160) Token > 166594924822352415786406422619018814804 changing ownership from / > 10.72.201.80 to /10.190.221.204 > INFO [main] 2012-09-10 08:00:21,855 StorageService.java (line 753) > JOINING: Starting to bootstrap... > INFO [CompactionExecutor:2] 2012-09-10 08:00:21,875 CompactionTask.java > (line 109) Compacting > [SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-1-Data.db'), > SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-3-Data.db'), > SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-4-Data.db'), > SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-2-Data.db')] > INFO [CompactionExecutor:2] 2012-09-10 08:00:21,979 CompactionTask.java > (line 221) Compacted to > [/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-5-Data.db,]. > 499 to 394 (~78% of original) bytes for 3 keys at 0.003997MB/s. Time: > 94ms. > INFO [Thread-4] 2012-09-10 08:00:22,070 StreamInSession.java (line 214) > Finished streaming session 1 from /10.72.102.61 > INFO [main] 2012-09-10 08:00:22,073 ColumnFamilyStore.java (line 643) > Enqueuing flush of Memtable-LocationInfo@30624226(77/96 serialized/live > bytes, 2 ops) > INFO [FlushWriter:2] 2012-09-10 08:00:22,074 Memtable.java (line 266) > Writing Memtable-LocationInfo@30624226(77/96 serialized/live bytes, 2 ops) > INFO [FlushWriter:2] 2012-09-10 08:00:22,082 Memtable.java (line 307) > Completed flushing > /mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-6-Data.db > (163 bytes) for commitlog position ReplayPosition(segmentId=6360937126265, > position=3854) > INFO [main] 2012-09-10 08:00:22,087 StorageService.java (line 1098) Node / > 10.190.221.204 state jump to normal > INFO [main] 2012-09-10 08:00:22,088 StorageService.java (line 666) > Bootstrap/Replace/Move completed! Now serving reads. > INFO [main] 2012-09-10 08:00:22,089 Mx4jTool.java (line 72) Will not load > MX4J, mx4j-tools.jar is not in the classpath > INFO [main] 2012-09-10 08:00:22,119 CassandraDaemon.java (line 124) > Binding thrift service to /10.190.221.204:9160 > INFO [main] 2012-09-10 08:00:22,122 CassandraDaemon.java (line 133) Using > TFastFramedTransport with a max frame size of 15728640 bytes. > INFO [main] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 160) Using > synchronous/threadpool thrift server on /10.190.221.204 : 9160 > INFO [Thread-5] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 212) > Listening for thrift clients... > INFO [RMI TCP Connection(4)-127.0.0.1] 2012-09-10 08:00:59,014 > ColumnFamilyStore.java (line 643) Enqueuing flush of > Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops) > INFO [FlushWriter:2] 2012-09-10 08:00:59,016 Memtable.java (line 266) > Writing Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops) > INFO [FlushWriter:2] 2012-09-10 08:00:59,024 Memtable.java (line 307) > Completed flushing > /mnt/cassandra/data/system/Versions/system-Versions-hd-1-Data.db (256 > bytes) for commitlog position ReplayPosition(segmentId=6360937126265, > position=3854) > INFO [GossipStage:1] 2012-09-10 08:01:23,036 Gossiper.java (line 850) > Node /10.72.201.80 is now part of the cluster > INFO [GossipStage:1] 2012-09-10 08:01:23,037 Gossiper.java (line 816) > InetAddress /10.72.201.80 is now UP > INFO [GossipStage:1] 2012-09-10 08:01:23,038 StorageService.java (line > 1126) Nodes /10.72.201.80 and /10.190.221.204 have the same token > 166594924822352415786406422619018814804. Ignoring /10.72.201.80 > INFO [GossipTasks:1] 2012-09-10 08:01:32,967 Gossiper.java (line 830) > InetAddress /10.72.201.80 is now dead. > INFO [GossipTasks:1] 2012-09-10 08:01:53,976 Gossiper.java (line 644) > FatClient /10.72.201.80 has been silent for 30000ms, removing from gossip > > >
