[ https://issues.apache.org/jira/browse/CASSANDRA-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Chia updated CASSANDRA-6753: ---------------------------------- Description: I was trying out the new release for several perf. improvements that I am very interested in. After upgrading my cassandra from 2.0.5 to the beta version, cassandra is stalled while init the column families. I might misconfigure something, but it seems it is suck in a loop. I added a couple debug statements, but, on second thought, I think I should just leave it to the experts... It's looping in the following over and over: {code:title=src/java/org/apache/cassandra/utils/memory/Pool.java#needsCleaning} 0 >= -858993472 && true && true {code} {code:title=Log} INFO [HeapSlabPoolCleaner] 2014-02-21 22:28:40,073 Keyspace.java:77 - java.lang.Thread.getStackTrace(Unknown Source), org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:77), org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:74), com.google.common.collect.Iterators$8.transform(Iterators.java:794), com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48), org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:2278), org.apache.cassandra.db.ColumnFamilyStore$FlushLargestColumnFamily.run(ColumnFamilyStore.java:1043), org.apache.cassandra.utils.memory.PoolCleanerThread.run(PoolCleanerThread.java:70) {code} They may be totally unrelated or a normal behavior. Let me know if there is any other info I should provide. was: I was trying out the new release for several perf. improvements that I am very interested in. After upgrading my cassandra from 2.0.5 to the beta version, cassandra is stalled while init the column families. I might misconfigure something, but it seems it is suck in a loop. I added a couple debug statements, but, on second thought, I think I should just leave it to the experts... It's looping in the following over and over: {code:title=src/java/org/apache/cassandra/utils/memory/Pool.java#needsCleaning} 0 >= -858993472 && true && true {code} {code:title=Log} INFO [HeapSlabPoolCleaner] 2014-02-21 22:28:40,073 Keyspace.java:77 - java.lang.Thread.getStackTrace(Unknown Source), org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:77), org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:74), com.google.common.collect.Iterators$8.transform(Iterators.java:794), com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48), org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:2278), org.apache.cassandra.db.ColumnFamilyStore$FlushLargestColumnFamily.run(ColumnFamilyStore.java:1043), org.apache.cassandra.utils.memory.PoolCleanerThread.run(PoolCleanerThread.java:70) {code} Let me know if there is any other info I should provide. > Cassandra2.1~beta1 Stall at Boot > -------------------------------- > > Key: CASSANDRA-6753 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6753 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Distributor ID: Ubuntu > Description: Ubuntu 12.04.3 LTS > Release: 12.04 > Codename: precise > AWS: i2.xlarge > {code} > INFO 22:34:40 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0 > INFO 22:34:40 Heap size: 12777553920/12777553920 > INFO 22:34:40 Code Cache Non-heap memory: init = 2555904(2496K) used = > 621632(607K) committed = 2555904(2496K) max = 50331648(49152K) > INFO 22:34:40 Par Eden Space Heap memory: init = 859045888(838912K) used = > 137447616(134226K) committed = 859045888(838912K) max = 859045888(838912K) > INFO 22:34:40 Par Survivor Space Heap memory: init = 107347968(104832K) used > = 0(0K) committed = 107347968(104832K) max = 107347968(104832K) > INFO 22:34:40 CMS Old Gen Heap memory: init = 11811160064(11534336K) used = > 1433816(1400K) committed = 11811160064(11534336K) max = 11811160064(11534336K) > INFO 22:34:40 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = > 18654512(18217K) committed = 21757952(21248K) max = 85983232(83968K) > INFO 22:34:40 Classpath: > /usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.6.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.0.13.jar:/usr/share/cassandra/lib/logback-core-1.0.13.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar:/usr/share/cassandra/CustomAgent.jar:/usr/share/cassandra/apache-cassandra-2.1.0~beta1.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.0~beta1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/jna.jar:/usr/share/cassandra/mx4j-tools.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.6.jar:/usr/share/cassandra/CustomAgent.jar:/usr/local/jcollectd/jcollectd.jar > {code} > {code:title=Node configuration} > [authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; > auto_snapshot=true; batchlog_replay_throttle_in_kb=1024; > cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; > cluster_name=sketchy_staging_test; column_index_size_in_kb=64; > commitlog_directory=/mnt/cassandra/commitlog; > commitlog_segment_size_in_mb=32; commitlog_sync=periodic; > commitlog_sync_period_in_ms=10000; compaction_preheat_key_cache=true; > compaction_throughput_mb_per_sec=64; concurrent_counter_writes=32; > concurrent_reads=128; concurrent_writes=128; counter_cache_save_period=7200; > counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; > cross_node_timeout=false; data_file_directories=[/mnt/cassandra/data]; > disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; > dynamic_snitch_reset_interval_in_ms=600000; > dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; > flush_directory=/mnt/cassandra/flush; hinted_handoff_enabled=true; > hinted_handoff_throttle_in_kb=1024; in_memory_compaction_limit_in_mb=64; > incremental_backups=false; index_summary_capacity_in_mb=null; > index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; > internode_compression=all; key_cache_save_period=14400; > key_cache_size_in_mb=1024; listen_address=10.9.163.158; > max_hint_window_in_ms=14400000; max_hints_delivery_threads=2; > memtable_cleanup_threshold=0.4; memtable_total_space_in_mb=2048; > native_transport_port=9042; num_tokens=256; > partitioner=org.apache.cassandra.dht.Murmur3Partitioner; > permissions_validity_in_ms=2000; preheat_kernel_page_cache=false; > range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; > request_scheduler=org.apache.cassandra.scheduler.NoScheduler; > request_timeout_in_ms=10000; row_cache_save_period=14400; > row_cache_size_in_mb=1024; rpc_address=0.0.0.0; rpc_keepalive=true; > rpc_port=9160; rpc_server_type=sync; > saved_caches_directory=/mnt/cassandra/cache; > seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, > parameters=[{seeds=10.71.141.38,10.218.142.35}]}]; > server_encryption_options=<REDACTED>; snapshot_before_compaction=false; > ssl_storage_port=7001; start_native_transport=true; start_rpc=true; > storage_port=7000; thrift_framed_transport_size_in_mb=15; > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; > trickle_fsync=true; trickle_fsync_interval_in_kb=10240; > truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000] > {code} > Reporter: David Chia > Fix For: 2.1 beta1 > > > I was trying out the new release for several perf. improvements that I am > very interested in. After upgrading my cassandra from 2.0.5 to the beta > version, cassandra is stalled while init the column families. > I might misconfigure something, but it seems it is suck in a loop. I added a > couple debug statements, but, on second thought, I think I should just leave > it to the experts... > It's looping in the following over and over: > {code:title=src/java/org/apache/cassandra/utils/memory/Pool.java#needsCleaning} > 0 >= -858993472 && true && true > {code} > {code:title=Log} > INFO [HeapSlabPoolCleaner] 2014-02-21 22:28:40,073 Keyspace.java:77 - > java.lang.Thread.getStackTrace(Unknown Source), > org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:77), > org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:74), > com.google.common.collect.Iterators$8.transform(Iterators.java:794), > com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48), > org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:2278), > org.apache.cassandra.db.ColumnFamilyStore$FlushLargestColumnFamily.run(ColumnFamilyStore.java:1043), > org.apache.cassandra.utils.memory.PoolCleanerThread.run(PoolCleanerThread.java:70) > {code} > They may be totally unrelated or a normal behavior. Let me know if there is > any other info I should provide. -- This message was sent by Atlassian JIRA (v6.1.5#6160)