I removed Priam and get the same picture.
What I do is- I added to cassandra-env.sh two lines and start cassandra. JVM_OPTS="$JVM_OPTS -Dcassandra.initial_token=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaba" JVM_OPTS="$JVM_OPTS -Dcassandra.replace_token=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaba" Then I can successfully run ring command Note: Ownership information does not include topology, please specify a keyspace. Address DC Rack Status State Load Owns Token Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaba]) 10.28.241.14 us-east 1a Up Normal 251.96 GB 33.33% Token(bytes[00000000000000000000000000000010]) 10.240.119.230 us-east 1b Up Normal 252.48 GB 33.33% Token(bytes[55555555555555555555555555555565]) 10.147.174.27 us-east 1c Up Normal 11.26 KB 33.33% Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaba]) It shows the current node as part of the ring, but it is empty. In data directory I can see only system key space. There is no any errors in log file. It just doen't stream data from other nodes. I can launch 1.1.6 but not 1.1.7 or higher. Any ideas what was changed in 1.1.7? Thank you, Andrey INFO [main] 2013-03-15 18:20:45,303 AbstractCassandraDaemon.java (line 101) Logging initialized INFO [main] 2013-03-15 18:20:45,309 AbstractCassandraDaemon.java (line 122) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_35 INFO [main] 2013-03-15 18:20:45,310 AbstractCassandraDaemon.java (line 123) Heap size: 1931476992/1931476992 INFO [main] 2013-03-15 18:20:45,311 AbstractCassandraDaemon.java (line 124) Classpath: /opt/apache-cassandra-1.1.10/bin/../conf:/opt/apache-cassandra-1.1.10/bin/../build/classes/main:/opt/apache-cassandra-1.1.10/bin/../build/classes/thrift:/opt/apache-cassandra-1.1.10/bin/../lib/antlr-3.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/apache-cassandra-1.1.10.jar:/opt/apache-cassandra-1.1.10/bin/../lib/apache-cassandra-clientutil-1.1.10.jar:/opt/apache-cassandra-1.1.10/bin/../lib/apache-cassandra-thrift-1.1.10.jar:/opt/apache-cassandra-1.1.10/bin/../lib/avro-1.4.0-fixes.jar:/opt/apache-cassandra-1.1.10/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/apache-cassandra-1.1.10/bin/../lib/commons-cli-1.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/commons-codec-1.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/commons-lang-2.4.jar:/opt/apache-cassandra-1.1.10/bin/../lib/compress-lzf-0.8.4.jar:/opt/apache-cassandra-1.1.10/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/opt/apache-cassandra-1.1.10/bin/../lib/guava-r08.jar:/opt/apache-cassandra-1.1.10/bin/../lib/high-scale-lib-1.1.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jackson-core-asl-1.9.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jackson-mapper-asl-1.9.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jamm-0.2.5.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jline-0.9.94.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jna.jar:/opt/apache-cassandra-1.1.10/bin/../lib/json-simple-1.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/libthrift-0.7.0.jar:/opt/apache-cassandra-1.1.10/bin/../lib/log4j-1.2.16.jar:/opt/apache-cassandra-1.1.10/bin/../lib/metrics-core-2.0.3.jar:/opt/apache-cassandra-1.1.10/bin/../lib/priam.jar:/opt/apache-cassandra-1.1.10/bin/../lib/servlet-api-2.5-20081211.jar:/opt/apache-cassandra-1.1.10/bin/../lib/slf4j-api-1.6.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/snakeyaml-1.6.jar:/opt/apache-cassandra-1.1.10/bin/../lib/snappy-java-1.0.4.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/snaptree-0.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jamm-0.2.5.jar INFO [main] 2013-03-15 18:20:47,406 CLibrary.java (line 111) JNA mlockall successful INFO [main] 2013-03-15 18:20:47,419 DatabaseDescriptor.java (line 123) Loading settings from file:/opt/apache-cassandra-1.1.10/conf/cassandra.yaml INFO [main] 2013-03-15 18:20:47,840 DatabaseDescriptor.java (line 182) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO [main] 2013-03-15 18:20:47,853 DatabaseDescriptor.java (line 246) Global memtable threshold is enabled at 614MB INFO [main] 2013-03-15 18:20:47,879 Ec2Snitch.java (line 66) EC2Snitch using region: us-east, zone: 1c. INFO [main] 2013-03-15 18:20:48,359 CacheService.java (line 96) Initializing key cache with capacity of 92 MBs. INFO [main] 2013-03-15 18:20:48,376 CacheService.java (line 107) Scheduling key cache save to each 14400 seconds (going to save all keys). INFO [main] 2013-03-15 18:20:48,377 CacheService.java (line 121) Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO [main] 2013-03-15 18:20:48,384 CacheService.java (line 133) Scheduling row cache save to each 0 seconds (going to save all keys). INFO [main] 2013-03-15 18:20:48,661 DatabaseDescriptor.java (line 509) Couldn't detect any schema definitions in local storage. INFO [main] 2013-03-15 18:20:48,676 DatabaseDescriptor.java (line 512) Found table data in data directories. Consider using the CLI to define your schema. INFO [main] 2013-03-15 18:20:48,716 CommitLog.java (line 124) No commitlog files found; skipping replay INFO [main] 2013-03-15 18:20:48,740 StorageService.java (line 433) Cassandra version: 1.1.10 INFO [main] 2013-03-15 18:20:48,740 StorageService.java (line 434) Thrift API version: 19.33.0 INFO [main] 2013-03-15 18:20:48,743 StorageService.java (line 435) CQL supported versions: 2.0.0,3.0.0-beta1 (default: 2.0.0) INFO [main] 2013-03-15 18:20:48,793 StorageService.java (line 465) Loading persisted ring state INFO [main] 2013-03-15 18:20:48,796 StorageService.java (line 546) Starting up server gossip INFO [main] 2013-03-15 18:20:48,809 ColumnFamilyStore.java (line 679) Enqueuing flush of Memtable-LocationInfo@523908271(124/155 serialized/live bytes, 3 ops) INFO [FlushWriter:1] 2013-03-15 18:20:48,810 Memtable.java (line 264) Writing Memtable-LocationInfo@523908271(124/155 serialized/live bytes, 3 ops) INFO [FlushWriter:1] 2013-03-15 18:20:48,858 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db (232 bytes) for commitlog position ReplayPosition(segmentId=1363371648611, position=585) INFO [main] 2013-03-15 18:20:48,873 Ec2Snitch.java (line 116) Ec2Snitch adding ApplicationState ec2region=us-east ec2zone=1c INFO [main] 2013-03-15 18:20:48,882 MessagingService.java (line 284) Starting Messaging Service on port 7000 INFO [main] 2013-03-15 18:20:48,890 StorageService.java (line 788) JOINING: waiting for ring information INFO [GossipStage:1] 2013-03-15 18:20:49,001 Gossiper.java (line 851) Node /10.240.119.230 is now part of the cluster INFO [GossipStage:1] 2013-03-15 18:20:49,002 Gossiper.java (line 817) InetAddress /10.240.119.230 is now UP INFO [GossipStage:1] 2013-03-15 18:20:49,004 ColumnFamilyStore.java (line 679) Enqueuing flush of Memtable-LocationInfo@64338076(52/65 serialized/live bytes, 2 ops) INFO [FlushWriter:1] 2013-03-15 18:20:49,005 Memtable.java (line 264) Writing Memtable-LocationInfo@64338076(52/65 serialized/live bytes, 2 ops) INFO [FlushWriter:1] 2013-03-15 18:20:49,015 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db (165 bytes) for commitlog position ReplayPosition(segmentId=1363371648611, position=768) INFO [GossipStage:1] 2013-03-15 18:20:49,019 Gossiper.java (line 851) Node /10.28.241.14 is now part of the cluster INFO [GossipStage:1] 2013-03-15 18:20:49,019 Gossiper.java (line 817) InetAddress /10.28.241.14 is now UP INFO [GossipStage:1] 2013-03-15 18:20:49,021 ColumnFamilyStore.java (line 679) Enqueuing flush of Memtable-LocationInfo@160194151(35/43 serialized/live bytes, 1 ops) INFO [FlushWriter:1] 2013-03-15 18:20:49,022 Memtable.java (line 264) Writing Memtable-LocationInfo@160194151(35/43 serialized/live bytes, 1 ops) INFO [FlushWriter:1] 2013-03-15 18:20:49,031 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db (89 bytes) for commitlog position ReplayPosition(segmentId=1363371648611, position=866) INFO [GossipStage:1] 2013-03-15 18:20:49,034 Gossiper.java (line 831) InetAddress /10.194.211.64 is now dead. INFO [main] 2013-03-15 18:21:18,893 StorageService.java (line 788) JOINING: schema complete INFO [main] 2013-03-15 18:21:18,894 StorageService.java (line 788) JOINING: waiting for pending range calculation INFO [main] 2013-03-15 18:21:18,894 StorageService.java (line 788) JOINING: calculation complete, ready to bootstrap INFO [main] 2013-03-15 18:22:18,895 StorageService.java (line 788) JOINING: Replacing a node with token: Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaba]) INFO [main] 2013-03-15 18:22:18,896 ColumnFamilyStore.java (line 679) Enqueuing flush of Memtable-LocationInfo@662811604(36/45 serialized/live bytes, 1 ops) INFO [FlushWriter:2] 2013-03-15 18:22:18,897 Memtable.java (line 264) Writing Memtable-LocationInfo@662811604(36/45 serialized/live bytes, 1 ops) INFO [FlushWriter:2] 2013-03-15 18:22:18,909 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db (87 bytes) for commitlog position ReplayPosition(segmentId=1363371648611, position=962) INFO [main] 2013-03-15 18:22:18,911 StorageService.java (line 788) JOINING: Starting to bootstrap... INFO [CompactionExecutor:3] 2013-03-15 18:22:18,929 CompactionTask.java (line 107) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db')] INFO [main] 2013-03-15 18:22:18,930 ColumnFamilyStore.java (line 679) Enqueuing flush of Memtable-LocationInfo@744919514(53/66 serialized/live bytes, 2 ops) INFO [FlushWriter:2] 2013-03-15 18:22:18,931 Memtable.java (line 264) Writing Memtable-LocationInfo@744919514(53/66 serialized/live bytes, 2 ops) INFO [FlushWriter:2] 2013-03-15 18:22:18,944 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db (163 bytes) for commitlog position ReplayPosition(segmentId=1363371648611, position=1143) INFO [main] 2013-03-15 18:22:18,969 StorageService.java (line 1133) Node ip-10-147-174-27.ec2.internal/10.147.174.27 state jump to normal INFO [main] 2013-03-15 18:22:18,969 StorageService.java (line 701) Bootstrap/Replace/Move completed! Now serving reads. INFO [main] 2013-03-15 18:22:19,011 CassandraDaemon.java (line 125) Binding thrift service to ip-10-147-174-27.ec2.internal/10.147.174.27:9160 INFO [main] 2013-03-15 18:22:19,015 CassandraDaemon.java (line 134) Using TFastFramedTransport with a max frame size of 15728640 bytes. INFO [main] 2013-03-15 18:22:19,019 CassandraDaemon.java (line 161) Using synchronous/threadpool thrift server on ip-10-147-174-27.ec2.internal/ 10.147.174.27 : 9160 INFO [Thread-6] 2013-03-15 18:22:19,020 CassandraDaemon.java (line 213) Listening for thrift clients... INFO [CompactionExecutor:3] 2013-03-15 18:22:19,031 CompactionTask.java (line 230) Compacted to [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db,]. 573 to 468 (~81% of original) bytes for 4 keys at 0.004649MB/s. Time: 96ms. On Fri, Mar 8, 2013 at 8:22 AM, aaron morton <aa...@thelastpickle.com>wrote: > If it does not have the schema check the logs for errors and ensure it is > actually part of the cluster. > > You may have better luck with Priam specific questions on > https://github.com/Netflix/Priam > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 7/03/2013, at 11:11 AM, Andrey Ilinykh <ailin...@gmail.com> wrote: > > Hello everybody! > > I used to run cassandra 1.1.5 with Priam. To replace dead node priam > launches cassandra with cassandra.replace_token property. It works smoothly > with 1.1.5. Couple days ago I moved to 1.1.10 and have a problem now. New > cassandra successfully starts, joins the ring but it doesn't see my > keyspaces. It doesn't try to stream data from other nodes. I see only > system keyspace. Any idea what is the difference between 1.1.5 and 1.1.10? > How am I supposed to replace dead node? > > Thank you, > Andrey > > >