[ https://issues.apache.org/jira/browse/CASSANDRA-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3417: -------------------------------------- Attachment: 3417-3.txt Attaching v3 which also adds synchronization in TokenMap.getTokenToEndpointMap() - both for bootstrapTokens and tokenToEndpointMap. (x.putAll(y) is not an atomic observation from the perspective of y, even if it is from the perspective of x) > InvocationTargetException ConcurrentModificationException at startup > -------------------------------------------------------------------- > > Key: CASSANDRA-3417 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3417 > Project: Cassandra > Issue Type: Bug > Affects Versions: 1.0.0 > Reporter: Joaquin Casares > Assignee: Jonathan Ellis > Priority: Minor > Fix For: 1.0.8 > > Attachments: 3417-2.txt, 3417-3.txt, 3417.txt > > > I was starting up the new DataStax AMI where the seed starts first and 34 > nodes would latch on together. So far things have been working decently for > launching, but right now I just got this during startup. > {CODE} > ubuntu@ip-10-40-190-143:~$ sudo cat /var/log/cassandra/output.log > INFO 09:24:38,453 JVM vendor/version: Java HotSpot(TM) 64-Bit Server > VM/1.6.0_26 > INFO 09:24:38,456 Heap size: 1936719872/1937768448 > INFO 09:24:38,457 Classpath: > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.3.jar:/usr/share/cassandra/apache-cassandra-1.0.0.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO 09:24:39,891 JNA mlockall successful > INFO 09:24:39,901 Loading settings from file:/etc/cassandra/cassandra.yaml > INFO 09:24:40,057 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 09:24:40,069 Global memtable threshold is enabled at 616MB > INFO 09:24:40,159 EC2Snitch using region: us-east, zone: 1d. > INFO 09:24:40,475 Creating new commitlog segment > /raid0/cassandra/commitlog/CommitLog-1319793880475.log > INFO 09:24:40,486 Couldn't detect any schema definitions in local storage. > INFO 09:24:40,486 Found table data in data directories. Consider using the > CLI to define your schema. > INFO 09:24:40,497 No commitlog files found; skipping replay > INFO 09:24:40,501 Cassandra version: 1.0.0 > INFO 09:24:40,502 Thrift API version: 19.18.0 > INFO 09:24:40,502 Loading persisted ring state > INFO 09:24:40,506 Starting up server gossip > INFO 09:24:40,529 Enqueuing flush of > Memtable-LocationInfo@1388314661(190/237 serialized/live bytes, 4 ops) > INFO 09:24:40,530 Writing Memtable-LocationInfo@1388314661(190/237 > serialized/live bytes, 4 ops) > INFO 09:24:40,600 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-1-Data.db (298 bytes) > INFO 09:24:40,613 Ec2Snitch adding ApplicationState ec2region=us-east > ec2zone=1d > INFO 09:24:40,621 Starting Messaging Service on /10.40.190.143:7000 > INFO 09:24:40,628 Joining: waiting for ring and schema information > INFO 09:24:43,389 InetAddress /10.194.29.156 is now dead. > INFO 09:24:43,391 InetAddress /10.85.11.38 is now dead. > INFO 09:24:43,392 InetAddress /10.34.42.28 is now dead. > INFO 09:24:43,393 InetAddress /10.77.63.49 is now dead. > INFO 09:24:43,394 InetAddress /10.194.22.191 is now dead. > INFO 09:24:43,395 InetAddress /10.34.74.58 is now dead. > INFO 09:24:43,395 Node /10.34.33.16 is now part of the cluster > INFO 09:24:43,396 InetAddress /10.34.33.16 is now UP > INFO 09:24:43,397 Enqueuing flush of Memtable-LocationInfo@1629818866(20/25 > serialized/live bytes, 1 ops) > INFO 09:24:43,398 Writing Memtable-LocationInfo@1629818866(20/25 > serialized/live bytes, 1 ops) > INFO 09:24:43,417 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-2-Data.db (74 bytes) > INFO 09:24:43,418 InetAddress /10.202.67.43 is now dead. > INFO 09:24:43,419 InetAddress /10.116.215.81 is now dead. > INFO 09:24:43,420 InetAddress /10.99.39.242 is now dead. > INFO 09:24:43,421 InetAddress /10.80.110.28 is now dead. > INFO 09:24:43,422 InetAddress /10.118.233.198 is now dead. > INFO 09:24:43,423 InetAddress /10.40.177.173 is now dead. > INFO 09:24:43,424 InetAddress /10.205.23.34 is now dead. > INFO 09:24:43,425 InetAddress /10.101.41.8 is now dead. > INFO 09:24:43,669 InetAddress /10.118.230.219 is now dead. > INFO 09:24:43,670 InetAddress /10.80.41.192 is now dead. > INFO 09:24:43,671 InetAddress /10.40.22.224 is now dead. > INFO 09:24:43,672 InetAddress /10.39.107.114 is now dead. > INFO 09:24:46,164 InetAddress /10.118.185.68 is now dead. > INFO 09:24:46,166 InetAddress /10.84.205.93 is now dead. > INFO 09:24:46,167 InetAddress /10.116.134.183 is now dead. > INFO 09:24:46,670 InetAddress /10.118.179.67 is now dead. > INFO 09:24:46,671 InetAddress /10.116.241.250 is now dead. > INFO 09:24:48,441 InetAddress /10.118.94.62 is now dead. > INFO 09:24:48,442 InetAddress /10.99.86.251 is now dead. > INFO 09:24:50,176 InetAddress /10.113.42.21 is now dead. > INFO 09:24:50,177 InetAddress /10.34.159.72 is now dead. > INFO 09:24:50,178 InetAddress /10.32.79.134 is now dead. > INFO 09:24:50,179 InetAddress /10.80.210.38 is now dead. > INFO 09:24:50,180 InetAddress /10.34.70.73 is now dead. > INFO 09:24:50,181 InetAddress /10.196.79.240 is now dead. > INFO 09:25:01,713 InetAddress /10.82.210.172 is now dead. > INFO 09:25:06,202 InetAddress /10.80.110.28 is now UP > INFO 09:25:06,908 InetAddress /10.99.39.242 is now UP > INFO 09:25:07,696 InetAddress /10.118.233.198 is now UP > INFO 09:25:07,697 InetAddress /10.205.23.34 is now UP > INFO 09:25:08,704 InetAddress /10.194.22.191 is now UP > INFO 09:25:08,705 InetAddress /10.40.177.173 is now UP > INFO 09:25:08,706 InetAddress /10.101.41.8 is now UP > INFO 09:25:09,489 InetAddress /10.202.67.43 is now UP > INFO 09:25:09,698 InetAddress /10.77.63.49 is now UP > INFO 09:25:10,628 Joining: getting bootstrap token > INFO 09:25:10,631 Enqueuing flush of Memtable-LocationInfo@1733057335(36/45 > serialized/live bytes, 1 ops) > INFO 09:25:10,631 Writing Memtable-LocationInfo@1733057335(36/45 > serialized/live bytes, 1 ops) > INFO 09:25:10,647 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-3-Data.db (87 bytes) > INFO 09:25:10,649 Joining: sleeping 30000 ms for pending range setup > INFO 09:25:10,689 InetAddress /10.85.11.38 is now UP > INFO 09:25:10,708 InetAddress /10.34.74.58 is now UP > INFO 09:25:10,912 InetAddress /10.194.29.156 is now UP > INFO 09:25:11,261 Applying migration bb843dd0-0146-11e1-0000-b877c09da5ff > Add keyspace: OpsCenter, rep > strategy:SimpleStrategy{org.apache.cassandra.config.CFMetaData@55e29b99[cfId=1000,ksName=OpsCenter,cfName=pdps,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=300.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@105585dc,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@5ec736e4[cfId=1004,ksName=OpsCenter,cfName=rollups86400,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=2,maxCompactionThreshold=8,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@68e4e358,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@b09dc35[cfId=1003,ksName=OpsCenter,cfName=rollups7200,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=2,maxCompactionThreshold=8,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@3458213c,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@5ee04fd[cfId=1002,ksName=OpsCenter,cfName=rollups300,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=16,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@4d898115,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@7e79b177[cfId=1005,ksName=OpsCenter,cfName=events,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=OpsCenter > raw event > storage,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=8,maxCompactionThreshold=12,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@67723c7f,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@540523be[cfId=1006,ksName=OpsCenter,cfName=events_timeline,cfType=Standard,comparator=org.apache.cassandra.db.marshal.LongType,subcolumncomparator=<null>,comment=OpsCenter > event > timelines,rowCacheSize=0.0,keyCacheSize=5.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=8,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=0,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@1d6dba0a,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@ed0f59e[cfId=1007,ksName=OpsCenter,cfName=settings,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=OpsCenter > > settings,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=8,maxCompactionThreshold=12,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@38ad5fab,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], > > org.apache.cassandra.config.CFMetaData@7e63f09e[cfId=1001,ksName=OpsCenter,cfName=rollups60,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@534a55e5,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}]}, > durable_writes: true > INFO 09:25:11,273 Enqueuing flush of > Memtable-Migrations@1767199109(12925/16156 serialized/live bytes, 1 ops) > INFO 09:25:11,273 Writing Memtable-Migrations@1767199109(12925/16156 > serialized/live bytes, 1 ops) > INFO 09:25:11,274 Enqueuing flush of Memtable-Schema@1616586953(5820/7275 > serialized/live bytes, 3 ops) > INFO 09:25:11,358 Completed flushing > /raid0/cassandra/data/system/Migrations-h-1-Data.db (12989 bytes) > INFO 09:25:11,358 Writing Memtable-Schema@1616586953(5820/7275 > serialized/live bytes, 3 ops) > INFO 09:25:11,390 Completed flushing > /raid0/cassandra/data/system/Schema-h-1-Data.db (5970 bytes) > INFO 09:25:11,727 InetAddress /10.116.215.81 is now UP > INFO 09:25:11,744 InetAddress /10.34.42.28 is now UP > INFO 09:25:11,750 InetAddress /10.40.22.224 is now UP > INFO 09:25:12,023 InetAddress /10.80.41.192 is now UP > INFO 09:25:12,712 InetAddress /10.39.107.114 is now UP > INFO 09:25:12,717 InetAddress /10.118.185.68 is now UP > INFO 09:25:12,721 InetAddress /10.116.134.183 is now UP > INFO 09:25:13,322 InetAddress /10.118.230.219 is now UP > INFO 09:25:13,632 InetAddress /10.84.205.93 is now UP > INFO 09:25:14,713 InetAddress /10.118.179.67 is now UP > INFO 09:25:14,717 InetAddress /10.116.241.250 is now UP > INFO 09:25:17,468 InetAddress /10.34.159.72 is now UP > INFO 09:25:17,476 InetAddress /10.118.94.62 is now UP > INFO 09:25:17,480 InetAddress /10.80.210.38 is now UP > INFO 09:25:17,716 InetAddress /10.32.79.134 is now UP > INFO 09:25:17,721 InetAddress /10.99.86.251 is now UP > INFO 09:25:18,717 InetAddress /10.196.79.240 is now UP > INFO 09:25:18,727 InetAddress /10.34.70.73 is now UP > INFO 09:25:19,596 InetAddress /10.113.42.21 is now UP > INFO 09:25:25,750 InetAddress /10.82.210.172 is now UP > INFO 09:25:37,743 Enqueuing flush of Memtable-LocationInfo@288976631(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:37,744 Writing Memtable-LocationInfo@288976631(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:37,764 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-4-Data.db (89 bytes) > INFO 09:25:37,773 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-1-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-3-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-4-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-2-Data.db')] > INFO 09:25:37,776 Enqueuing flush of Memtable-LocationInfo@1950702248(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:37,777 Writing Memtable-LocationInfo@1950702248(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:37,821 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-5-Data.db (89 bytes) > INFO 09:25:37,869 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-6-Data.db,]. 548 to 443 (~80% > of original) bytes for 3 keys at 0.006500MB/s. Time: 65ms. > INFO 09:25:38,740 Enqueuing flush of Memtable-LocationInfo@92917455(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:38,740 Writing Memtable-LocationInfo@92917455(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:38,757 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-8-Data.db (89 bytes) > INFO 09:25:38,766 Enqueuing flush of Memtable-LocationInfo@1096488363(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:38,767 Writing Memtable-LocationInfo@1096488363(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:38,814 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-9-Data.db (89 bytes) > INFO 09:25:38,816 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-6-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-9-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-8-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-5-Data.db')] > INFO 09:25:38,823 Enqueuing flush of Memtable-LocationInfo@1734564525(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:38,823 Writing Memtable-LocationInfo@1734564525(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:38,893 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-10-Data.db (89 bytes) > INFO 09:25:38,916 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-11-Data.db,]. 710 to 548 (~77% > of original) bytes for 3 keys at 0.005226MB/s. Time: 100ms. > INFO 09:25:39,538 Enqueuing flush of Memtable-LocationInfo@811507066(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,539 Writing Memtable-LocationInfo@811507066(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,555 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-13-Data.db (89 bytes) > INFO 09:25:39,578 Enqueuing flush of Memtable-LocationInfo@1125690366(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,578 Writing Memtable-LocationInfo@1125690366(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,594 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-14-Data.db (89 bytes) > INFO 09:25:39,596 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-11-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-10-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-14-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-13-Data.db')] > INFO 09:25:39,613 Enqueuing flush of Memtable-LocationInfo@1870148830(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,614 Writing Memtable-LocationInfo@1870148830(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,652 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-16-Data.db (89 bytes) > INFO 09:25:39,692 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-15-Data.db,]. 815 to 653 (~80% > of original) bytes for 3 keys at 0.006487MB/s. Time: 96ms. > INFO 09:25:39,731 Enqueuing flush of Memtable-LocationInfo@1279866611(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,731 Writing Memtable-LocationInfo@1279866611(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:39,747 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-18-Data.db (89 bytes) > INFO 09:25:40,649 Starting to bootstrap... > INFO 09:25:40,701 Finished streaming session 304272969286 from /10.205.23.34 > INFO 09:25:40,703 Enqueuing flush of Memtable-LocationInfo@1868577756(53/66 > serialized/live bytes, 2 ops) > INFO 09:25:40,703 Writing Memtable-LocationInfo@1868577756(53/66 > serialized/live bytes, 2 ops) > INFO 09:25:40,721 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-19-Data.db (163 bytes) > INFO 09:25:40,722 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-19-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-15-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-18-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-16-Data.db')] > INFO 09:25:40,726 Node /10.40.190.143 state jump to normal > INFO 09:25:40,734 Enqueuing flush of Memtable-LocationInfo@641287650(35/43 > serialized/live bytes, 1 ops) > INFO 09:25:40,735 Writing Memtable-LocationInfo@641287650(35/43 > serialized/live bytes, 1 ops) > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) > Caused by: java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) > at java.util.HashMap$EntryIterator.next(HashMap.java:834) > at java.util.HashMap$EntryIterator.next(HashMap.java:832) > at > com.google.common.collect.AbstractBiMap$EntrySet$1.next(AbstractBiMap.java:301) > at > com.google.common.collect.AbstractBiMap$EntrySet$1.next(AbstractBiMap.java:293) > at > org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:1127) > at > org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:1084) > at > org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:920) > at > org.apache.cassandra.service.StorageService.onChange(StorageService.java:805) > at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:880) > at > org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1027) > at > org.apache.cassandra.service.StorageService.setToken(StorageService.java:226) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:573) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:460) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:381) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:215) > at > org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:238) > ... 5 more > Cannot load daemon > Service exit with a return value of 3 > INFO 09:35:35,156 JVM vendor/version: Java HotSpot(TM) 64-Bit Server > VM/1.6.0_26 > INFO 09:35:35,159 Heap size: 1936719872/1937768448 > INFO 09:35:35,160 Classpath: > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.3.jar:/usr/share/cassandra/apache-cassandra-1.0.0.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO 09:35:36,626 JNA mlockall successful > INFO 09:35:36,636 Loading settings from file:/etc/cassandra/cassandra.yaml > INFO 09:35:36,757 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 09:35:36,769 Global memtable threshold is enabled at 616MB > INFO 09:35:36,811 EC2Snitch using region: us-east, zone: 1d. > INFO 09:35:37,030 Opening /raid0/cassandra/data/system/Schema-h-1 (5970 > bytes) > INFO 09:35:37,067 Opening /raid0/cassandra/data/system/Migrations-h-1 (12989 > bytes) > INFO 09:35:37,075 Opening /raid0/cassandra/data/system/LocationInfo-h-19 > (163 bytes) > INFO 09:35:37,075 Opening /raid0/cassandra/data/system/LocationInfo-h-18 (89 > bytes) > INFO 09:35:37,083 Opening /raid0/cassandra/data/system/LocationInfo-h-15 > (653 bytes) > INFO 09:35:37,085 Opening /raid0/cassandra/data/system/LocationInfo-h-16 (89 > bytes) > INFO 09:35:37,131 Loading schema version bb843dd0-0146-11e1-0000-b877c09da5ff > INFO 09:35:37,372 Creating new commitlog segment > /raid0/cassandra/commitlog/CommitLog-1319794537372.log > INFO 09:35:37,384 Replaying > /raid0/cassandra/commitlog/CommitLog-1319793880475.log > INFO 09:35:37,416 Finished reading > /raid0/cassandra/commitlog/CommitLog-1319793880475.log > INFO 09:35:37,422 Enqueuing flush of Memtable-events@1830423861(164/205 > serialized/live bytes, 5 ops) > INFO 09:35:37,423 Writing Memtable-events@1830423861(164/205 serialized/live > bytes, 5 ops) > INFO 09:35:37,424 Enqueuing flush of Memtable-Versions@817138449(83/103 > serialized/live bytes, 3 ops) > INFO 09:35:37,472 Completed flushing > /raid0/cassandra/data/OpsCenter/events-h-1-Data.db (230 bytes) > INFO 09:35:37,479 Writing Memtable-Versions@817138449(83/103 serialized/live > bytes, 3 ops) > INFO 09:35:37,497 Completed flushing > /raid0/cassandra/data/system/Versions-h-1-Data.db (247 bytes) > INFO 09:35:37,497 Log replay complete, 4 replayed mutations > INFO 09:35:37,509 Cassandra version: 1.0.0 > INFO 09:35:37,510 Thrift API version: 19.18.0 > INFO 09:35:37,510 Loading persisted ring state > INFO 09:35:37,528 Starting up server gossip > INFO 09:35:37,530 Enqueuing flush of Memtable-LocationInfo@1655441108(29/36 > serialized/live bytes, 1 ops) > INFO 09:35:37,530 Writing Memtable-LocationInfo@1655441108(29/36 > serialized/live bytes, 1 ops) > INFO 09:35:37,554 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-20-Data.db (80 bytes) > INFO 09:35:37,555 Ec2Snitch adding ApplicationState ec2region=us-east > ec2zone=1d > INFO 09:35:37,562 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-16-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-18-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-19-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-20-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-15-Data.db')] > INFO 09:35:37,566 Starting Messaging Service on /10.40.190.143:7000 > INFO 09:35:37,592 Using saved token 19444706681196483626478548996101040654 > INFO 09:35:37,593 Enqueuing flush of Memtable-LocationInfo@995684858(53/66 > serialized/live bytes, 2 ops) > INFO 09:35:37,593 Writing Memtable-LocationInfo@995684858(53/66 > serialized/live bytes, 2 ops) > INFO 09:35:37,616 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-22-Data.db (163 bytes) > INFO 09:35:37,620 Node /10.40.190.143 state jump to normal > INFO 09:35:37,639 Bootstrap/Replace/Move completed! Now serving reads. > INFO 09:35:37,640 Will not load MX4J, mx4j-tools.jar is not in the classpath > INFO 09:35:37,684 Binding thrift service to /0.0.0.0:9160 > INFO 09:35:37,687 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-21-Data.db,]. 1,074 to 799 > (~74% of original) bytes for 4 keys at 0.007620MB/s. Time: 100ms. > INFO 09:35:37,688 Using TFastFramedTransport with a max frame size of > 15728640 bytes. > INFO 09:35:37,692 Using synchronous/threadpool thrift server on /0.0.0.0 : > 9160 > INFO 09:35:37,695 Listening for thrift clients... > INFO 09:35:37,706 Node /10.118.230.219 is now part of the cluster > INFO 09:35:37,707 InetAddress /10.118.230.219 is now UP > INFO 09:35:37,708 Enqueuing flush of Memtable-LocationInfo@2035487037(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,709 Writing Memtable-LocationInfo@2035487037(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,725 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-24-Data.db (89 bytes) > INFO 09:35:37,726 Node /10.34.42.28 is now part of the cluster > INFO 09:35:37,727 InetAddress /10.34.42.28 is now UP > INFO 09:35:37,729 Enqueuing flush of Memtable-LocationInfo@321887181(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,729 Writing Memtable-LocationInfo@321887181(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,747 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-25-Data.db (89 bytes) > INFO 09:35:37,748 Node /10.77.63.49 has restarted, now UP > INFO 09:35:37,748 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-24-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-22-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-25-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-21-Data.db')] > INFO 09:35:37,748 InetAddress /10.77.63.49 is now UP > INFO 09:35:37,749 Node /10.77.63.49 state jump to normal > INFO 09:35:37,750 Node /10.34.70.73 is now part of the cluster > INFO 09:35:37,750 InetAddress /10.34.70.73 is now UP > INFO 09:35:37,752 Enqueuing flush of Memtable-LocationInfo@1354749546(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,752 Writing Memtable-LocationInfo@1354749546(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,789 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-26-Data.db,]. 1,140 to 877 > (~76% of original) bytes for 4 keys at 0.020399MB/s. Time: 41ms. > INFO 09:35:37,801 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-27-Data.db (89 bytes) > INFO 09:35:37,801 Node /10.99.86.251 is now part of the cluster > INFO 09:35:37,802 InetAddress /10.99.86.251 is now UP > INFO 09:35:37,803 Enqueuing flush of Memtable-LocationInfo@793374785(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,804 Writing Memtable-LocationInfo@793374785(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,825 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-29-Data.db (89 bytes) > INFO 09:35:37,826 Node /10.202.67.43 has restarted, now UP > INFO 09:35:37,827 InetAddress /10.202.67.43 is now UP > INFO 09:35:37,827 Node /10.202.67.43 state jump to normal > INFO 09:35:37,828 Node /10.116.134.183 is now part of the cluster > INFO 09:35:37,828 InetAddress /10.116.134.183 is now UP > INFO 09:35:37,829 Enqueuing flush of Memtable-LocationInfo@1728699027(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,830 Writing Memtable-LocationInfo@1728699027(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,850 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-30-Data.db (89 bytes) > INFO 09:35:37,852 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-30-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-27-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-26-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-29-Data.db')] > INFO 09:35:37,853 Node /10.118.94.62 is now part of the cluster > INFO 09:35:37,853 InetAddress /10.118.94.62 is now UP > INFO 09:35:37,855 Enqueuing flush of Memtable-LocationInfo@2001229122(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,855 Writing Memtable-LocationInfo@2001229122(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,885 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-31-Data.db (89 bytes) > INFO 09:35:37,886 Node /10.116.215.81 is now part of the cluster > INFO 09:35:37,887 InetAddress /10.116.215.81 is now UP > INFO 09:35:37,888 Enqueuing flush of Memtable-LocationInfo@1748800276(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,888 Writing Memtable-LocationInfo@1748800276(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,909 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-33-Data.db (89 bytes) > INFO 09:35:37,910 Node /10.80.110.28 has restarted, now UP > INFO 09:35:37,911 InetAddress /10.80.110.28 is now UP > INFO 09:35:37,911 Node /10.80.110.28 state jump to normal > INFO 09:35:37,912 Node /10.80.210.38 is now part of the cluster > INFO 09:35:37,912 InetAddress /10.80.210.38 is now UP > INFO 09:35:37,914 Enqueuing flush of Memtable-LocationInfo@1761382005(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,914 Writing Memtable-LocationInfo@1761382005(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,925 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-32-Data.db,]. 1,144 to 982 > (~85% of original) bytes for 4 keys at 0.014190MB/s. Time: 66ms. > INFO 09:35:37,927 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-35-Data.db (89 bytes) > INFO 09:35:37,928 Node /10.40.177.173 has restarted, now UP > INFO 09:35:37,929 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-31-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-32-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-33-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-35-Data.db')] > INFO 09:35:37,929 InetAddress /10.40.177.173 is now UP > INFO 09:35:37,929 Node /10.40.177.173 state jump to normal > INFO 09:35:37,930 Node /10.101.41.8 has restarted, now UP > INFO 09:35:37,931 InetAddress /10.101.41.8 is now UP > INFO 09:35:37,931 Node /10.101.41.8 state jump to normal > INFO 09:35:37,931 Node /10.205.23.34 has restarted, now UP > INFO 09:35:37,932 InetAddress /10.205.23.34 is now UP > INFO 09:35:37,932 Node /10.205.23.34 state jump to normal > INFO 09:35:37,933 Node /10.118.185.68 is now part of the cluster > INFO 09:35:37,933 InetAddress /10.118.185.68 is now UP > INFO 09:35:37,934 Enqueuing flush of Memtable-LocationInfo@260440278(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,935 Writing Memtable-LocationInfo@260440278(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,970 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-36-Data.db (89 bytes) > INFO 09:35:37,971 Node /10.116.241.250 is now part of the cluster > INFO 09:35:37,972 InetAddress /10.116.241.250 is now UP > INFO 09:35:37,973 Enqueuing flush of Memtable-LocationInfo@768673839(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:37,974 Writing Memtable-LocationInfo@768673839(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,003 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-38-Data.db (89 bytes) > INFO 09:35:38,004 Node /10.113.42.21 is now part of the cluster > INFO 09:35:38,005 InetAddress /10.113.42.21 is now UP > INFO 09:35:38,007 Enqueuing flush of Memtable-LocationInfo@1610335061(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,008 Writing Memtable-LocationInfo@1610335061(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,014 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-37-Data.db,]. 1,249 to 1,087 > (~87% of original) bytes for 4 keys at 0.012196MB/s. Time: 85ms. > INFO 09:35:38,024 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-40-Data.db (89 bytes) > INFO 09:35:38,024 Node /10.194.29.156 is now part of the cluster > INFO 09:35:38,025 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-37-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-40-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-36-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-38-Data.db')] > INFO 09:35:38,025 InetAddress /10.194.29.156 is now UP > INFO 09:35:38,026 Enqueuing flush of Memtable-LocationInfo@1625488363(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,027 Writing Memtable-LocationInfo@1625488363(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,042 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-41-Data.db (89 bytes) > INFO 09:35:38,043 Node /10.85.11.38 has restarted, now UP > INFO 09:35:38,044 InetAddress /10.85.11.38 is now UP > INFO 09:35:38,044 Node /10.85.11.38 state jump to normal > INFO 09:35:38,045 Node /10.34.159.72 is now part of the cluster > INFO 09:35:38,046 InetAddress /10.34.159.72 is now UP > INFO 09:35:38,047 Enqueuing flush of Memtable-LocationInfo@747881713(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,048 Writing Memtable-LocationInfo@747881713(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,065 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-42-Data.db (89 bytes) > INFO 09:35:38,067 Node /10.194.22.191 is now part of the cluster > INFO 09:35:38,067 InetAddress /10.194.22.191 is now UP > INFO 09:35:38,069 Enqueuing flush of Memtable-LocationInfo@709926392(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,069 Writing Memtable-LocationInfo@709926392(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,092 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-44-Data.db (89 bytes) > INFO 09:35:38,093 Node /10.34.74.58 is now part of the cluster > INFO 09:35:38,097 InetAddress /10.34.74.58 is now UP > INFO 09:35:38,098 Enqueuing flush of Memtable-LocationInfo@1356841826(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,099 Writing Memtable-LocationInfo@1356841826(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,105 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-43-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-41-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-44-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-42-Data.db')] > INFO 09:35:38,106 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-43-Data.db,]. 1,354 to 1,192 > (~88% of original) bytes for 4 keys at 0.014034MB/s. Time: 81ms. > INFO 09:35:38,144 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-46-Data.db (89 bytes) > INFO 09:35:38,145 Node /10.40.22.224 is now part of the cluster > INFO 09:35:38,146 InetAddress /10.40.22.224 is now UP > INFO 09:35:38,147 Enqueuing flush of Memtable-LocationInfo@422797318(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,148 Writing Memtable-LocationInfo@422797318(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,155 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-47-Data.db,]. 1,459 to 1,297 > (~88% of original) bytes for 4 keys at 0.024738MB/s. Time: 50ms. > INFO 09:35:38,164 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-49-Data.db (89 bytes) > INFO 09:35:38,165 Node /10.32.79.134 is now part of the cluster > INFO 09:35:38,166 InetAddress /10.32.79.134 is now UP > INFO 09:35:38,167 Enqueuing flush of Memtable-LocationInfo@1455093129(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,168 Writing Memtable-LocationInfo@1455093129(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,199 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-50-Data.db (89 bytes) > INFO 09:35:38,200 Node /10.118.179.67 is now part of the cluster > INFO 09:35:38,200 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-50-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-47-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-49-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-46-Data.db')] > INFO 09:35:38,200 InetAddress /10.118.179.67 is now UP > INFO 09:35:38,202 Enqueuing flush of Memtable-LocationInfo@1105436908(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,202 Writing Memtable-LocationInfo@1105436908(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,248 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-51-Data.db (89 bytes) > INFO 09:35:38,249 Node /10.84.205.93 is now part of the cluster > INFO 09:35:38,249 InetAddress /10.84.205.93 is now UP > INFO 09:35:38,251 Enqueuing flush of Memtable-LocationInfo@1306980591(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,251 Writing Memtable-LocationInfo@1306980591(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,262 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-52-Data.db,]. 1,564 to 1,402 > (~89% of original) bytes for 4 keys at 0.021919MB/s. Time: 61ms. > INFO 09:35:38,294 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-54-Data.db (89 bytes) > INFO 09:35:38,294 Node /10.34.33.16 has restarted, now UP > INFO 09:35:38,295 InetAddress /10.34.33.16 is now UP > INFO 09:35:38,296 Node /10.34.33.16 state jump to normal > INFO 09:35:38,296 Node /10.39.107.114 is now part of the cluster > INFO 09:35:38,297 InetAddress /10.39.107.114 is now UP > INFO 09:35:38,298 Enqueuing flush of Memtable-LocationInfo@1038389338(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,299 Writing Memtable-LocationInfo@1038389338(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,311 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-55-Data.db (89 bytes) > INFO 09:35:38,312 Node /10.196.79.240 is now part of the cluster > INFO 09:35:38,312 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-52-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-55-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-54-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-51-Data.db')] > INFO 09:35:38,313 InetAddress /10.196.79.240 is now UP > INFO 09:35:38,314 Enqueuing flush of Memtable-LocationInfo@1850278722(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,315 Writing Memtable-LocationInfo@1850278722(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,354 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-56-Data.db (89 bytes) > INFO 09:35:38,355 Node /10.99.39.242 has restarted, now UP > INFO 09:35:38,356 InetAddress /10.99.39.242 is now UP > INFO 09:35:38,356 Node /10.99.39.242 state jump to normal > INFO 09:35:38,357 Node /10.118.233.198 has restarted, now UP > INFO 09:35:38,358 InetAddress /10.118.233.198 is now UP > INFO 09:35:38,358 Node /10.118.233.198 state jump to normal > INFO 09:35:38,359 Node /10.82.210.172 is now part of the cluster > INFO 09:35:38,359 InetAddress /10.82.210.172 is now UP > INFO 09:35:38,364 Enqueuing flush of Memtable-LocationInfo@786665924(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,364 Writing Memtable-LocationInfo@786665924(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,439 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-58-Data.db (89 bytes) > INFO 09:35:38,440 Node /10.80.41.192 is now part of the cluster > INFO 09:35:38,440 InetAddress /10.80.41.192 is now UP > INFO 09:35:38,442 Enqueuing flush of Memtable-LocationInfo@1647844754(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,442 Writing Memtable-LocationInfo@1647844754(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,451 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-57-Data.db,]. 1,669 to 1,515 > (~90% of original) bytes for 4 keys at 0.010470MB/s. Time: 138ms. > INFO 09:35:38,459 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-60-Data.db (89 bytes) > INFO 09:35:38,459 Node /10.76.243.129 is now part of the cluster > INFO 09:35:38,460 Compacting > [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-56-Data.db'), > > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-58-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-57-Data.db'), > SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-60-Data.db')] > INFO 09:35:38,460 InetAddress /10.76.243.129 is now UP > INFO 09:35:38,462 Enqueuing flush of Memtable-LocationInfo@585652261(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,462 Writing Memtable-LocationInfo@585652261(35/43 > serialized/live bytes, 1 ops) > INFO 09:35:38,478 Completed flushing > /raid0/cassandra/data/system/LocationInfo-h-61-Data.db (89 bytes) > INFO 09:35:38,486 Node /10.34.42.28 state jump to normal > INFO 09:35:38,487 Node /10.34.70.73 state jump to normal > INFO 09:35:38,488 Node /10.99.86.251 state jump to normal > INFO 09:35:38,489 Node /10.118.94.62 state jump to normal > INFO 09:35:38,489 Node /10.80.110.28 state jump to normal > INFO 09:35:38,490 Node /10.80.210.38 state jump to normal > INFO 09:35:38,491 Node /10.40.177.173 state jump to normal > INFO 09:35:38,493 Node /10.101.41.8 state jump to normal > INFO 09:35:38,493 Node /10.113.42.21 state jump to normal > INFO 09:35:38,494 Node /10.85.11.38 state jump to normal > INFO 09:35:38,495 Node /10.34.159.72 state jump to normal > INFO 09:35:38,496 Node /10.34.74.58 state jump to normal > INFO 09:35:38,497 Node /10.84.205.93 state jump to normal > INFO 09:35:38,497 Node /10.118.179.67 state jump to normal > INFO 09:35:38,498 Node /10.34.33.16 state jump to normal > INFO 09:35:38,499 Node /10.196.79.240 state jump to normal > INFO 09:35:38,500 Node /10.118.233.198 state jump to normal > INFO 09:35:38,501 Node /10.80.41.192 state jump to normal > INFO 09:35:38,502 Node /10.76.243.129 state jump to normal > INFO 09:35:38,508 Node /10.118.185.68 state jump to normal > INFO 09:35:38,524 Node /10.118.230.219 state jump to normal > INFO 09:35:38,536 Compacted to > [/raid0/cassandra/data/system/LocationInfo-h-62-Data.db,]. 1,782 to 1,620 > (~90% of original) bytes for 4 keys at 0.020328MB/s. Time: 76ms. > INFO 09:35:38,537 Node /10.80.110.28 state jump to normal > INFO 09:35:38,537 Node /10.40.177.173 state jump to normal > INFO 09:35:38,538 Node /10.101.41.8 state jump to normal > INFO 09:35:38,539 Node /10.116.241.250 state jump to normal > INFO 09:35:38,540 Node /10.194.29.156 state jump to normal > INFO 09:35:38,540 Node /10.34.74.58 state jump to normal > INFO 09:35:38,541 Node /10.40.22.224 state jump to normal > INFO 09:35:38,542 Node /10.32.79.134 state jump to normal > INFO 09:35:38,543 Node /10.39.107.114 state jump to normal > INFO 09:35:38,543 Node /10.99.39.242 state jump to normal > INFO 09:35:38,550 Node /10.77.63.49 state jump to normal > INFO 09:35:38,550 Node /10.34.42.28 state jump to normal > INFO 09:35:38,551 Node /10.116.134.183 state jump to normal > INFO 09:35:38,553 Node /10.76.243.129 state jump to normal > INFO 09:35:38,557 Node /10.202.67.43 state jump to normal > INFO 09:35:38,558 Node /10.118.94.62 state jump to normal > INFO 09:35:38,562 Node /10.116.215.81 state jump to normal > INFO 09:35:38,563 Node /10.80.210.38 state jump to normal > INFO 09:35:38,564 Node /10.205.23.34 state jump to normal > INFO 09:35:38,565 Node /10.39.107.114 state jump to normal > {CODE} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira