After trying every possible combination of parameters, config and the rest,
I ended up downgrading the new node from 1.1.11 to 1.1.2 to match the
existing 3 nodes. And that solved the issue immediately : the schema was
propagated and the node started handling reads & writes.
2013/5/3 Philippe <watche...@gmail.com>
> Unfortunately not, I've moved on to trying to add the nodes the current
> cluster and then decommission the "old" ones.
>
> But even that is not working, this is the strangest of things : while
> trying to add a new node, I
> - set its token to an existing value+1
> - ensure the yaml (clutser name, partitionner, etc) are the same
> - verified I can connect to the :7000 ports across machines
> - cleared the data and commitlog directory
>
> when I start the node, it goes through the bootstrap process but never
> "imports" the schema from the cluster (the data/ directory stays empty,
> permissions are correct) and I get errors when some reads come in after the
> bootstrap completes. I've tried restarting the node with -D replace.token
> The log is below. I've been at it all day so I've been adding the node &
> decommissioning it, assuming that would clear any state in the cluster.
>
> INFO [main] 2013-05-03 21:10:22,126 StorageService.java (line 788)
> JOINING: waiting for ring information
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,421 Gossiper.java (line 838)
> InetAddress /172.16.0.42 is now dead.
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,427 StorageService.java (line
> 1303) Removing token Token(bytes[03abaa3000000000000001]) for /172.16.0.42
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,429 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@444462602(47/58
> serialized/live bytes, 2 ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,430 Memtable.java (line 264)
> Writing Memtable-LocationInfo@444462602(47/58 serialized/live bytes, 2
> ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,542 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db
> (160 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=769)
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,544 Gossiper.java (line 858)
> Node /{other_node_A} is now part of the cluster
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,545 Gossiper.java (line 824)
> InetAddress /{other_node_A} is now UP
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,547 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@1290155526(30/37
> serialized/live bytes, 1 ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,548 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1290155526(30/37 serialized/live bytes, 1
> ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,662 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db
> (84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=862)
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,664 Gossiper.java (line 858)
> Node /{other_node_B} is now part of the cluster
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,665 Gossiper.java (line 824)
> InetAddress /{other_node_B} is now UP
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,666 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@899933215(30/37
> serialized/live bytes, 1 ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,667 Memtable.java (line 264)
> Writing Memtable-LocationInfo@899933215(30/37 serialized/live bytes, 1
> ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,779 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db
> (84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=955)
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,781 Gossiper.java (line 858)
> Node /{other_node_C} is now part of the cluster
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,782 Gossiper.java (line 824)
> InetAddress /{other_node_C} is now UP
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,784 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@1542631196(30/37
> serialized/live bytes, 1 ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,785 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1542631196(30/37 serialized/live bytes, 1
> ops)
>
> INFO [CompactionExecutor:4] 2013-05-03 21:10:26,789 CompactionTask.java
> (line 107) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db')]
>
> INFO [FlushWriter:1] 2013-05-03 21:10:26,939 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db
> (84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=1048)
>
> INFO [GossipStage:1] 2013-05-03 21:10:26,942 Gossiper.java (line 838)
> InetAddress /172.16.0.39 is now dead.
>
> INFO [CompactionExecutor:4] 2013-05-03 21:10:26,986 CompactionTask.java
> (line 232) Compacted to
> [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db,].
> 566 to 458 (~80% of original) bytes for 4 keys at 0.002287MB/s. Time:
> 191ms.
>
> INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
> JOINING: schema complete
>
> INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
> JOINING: waiting for pending range calculation
>
> INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
> JOINING: calculation complete, ready to bootstrap
>
> INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
> JOINING: getting bootstrap token
>
> INFO [main] 2013-05-03 21:10:52,135 ColumnFamilyStore.java (line 674)
> Enqueuing flush of Memtable-LocationInfo@1692466348(31/38 serialized/live
> bytes, 1 ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:52,136 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1692466348(31/38 serialized/live bytes, 1
> ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:10:52,252 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db
> (82 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=1139)
>
> INFO [main] 2013-05-03 21:10:52,254 StorageService.java (line 788)
> JOINING: sleeping 30000 ms for pending range setup
>
> INFO [main] 2013-05-03 21:11:22,255 StorageService.java (line 788)
> JOINING: Starting to bootstrap...
>
> INFO [main] 2013-05-03 21:11:22,272 ColumnFamilyStore.java (line 674)
> Enqueuing flush of Memtable-LocationInfo@1458621244(48/60 serialized/live
> bytes, 2 ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:11:22,273 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1458621244(48/60 serialized/live bytes, 2
> ops)
>
> INFO [FlushWriter:1] 2013-05-03 21:11:22,386 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db
> (158 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=1315)
>
> INFO [CompactionExecutor:8] 2013-05-03 21:11:22,389 CompactionTask.java
> (line 107) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db')]
>
> INFO [main] 2013-05-03 21:11:22,396 StorageService.java (line 1133) Node /
> 5.135.140.99 state jump to normal
>
> INFO [main] 2013-05-03 21:11:22,397 StorageService.java (line 701)
> Bootstrap/Replace/Move completed! Now serving reads.
>
> INFO [main] 2013-05-03 21:11:22,415 CassandraDaemon.java (line 125)
> Binding thrift service to /0.0.0.0:9160
>
> INFO [main] 2013-05-03 21:11:22,417 CassandraDaemon.java (line 134) Using
> TFastFramedTransport with a max frame size of 15728640 bytes.
>
> INFO [main] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 161) Using
> synchronous/threadpool thrift server on /0.0.0.0 : 9160
>
> INFO [Thread-9] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 213)
> Listening for thrift clients...
>
> INFO [CompactionExecutor:8] 2013-05-03 21:11:22,542 CompactionTask.java
> (line 232) Compacted to
> [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-9-Data.db,].
> 782 to 489 (~62% of original) bytes for 4 keys at 0.003088MB/s. Time:
> 151ms.
>
>
