[ 
https://issues.apache.org/jira/browse/CASSANDRA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732098#comment-13732098
 ] 

Aaron Daubman commented on CASSANDRA-5605:
------------------------------------------

Apologies if this isn't directly relevant, but I seem to be experiencing the 
same issue using 1.2.8 launched via ccm for integration testing. One 
differentiating feature here is that this happened the first time the node was 
ever brought up (0 data). The integration tests attempting to use the ccm 
cluster never completed due to this hang, but all they do is test creation of a 
fairly simple schema and then attempt to write and then read back a single row.

Here's what was in the log:
 INFO [main] 2013-08-07 14:56:46,763 CassandraDaemon.java (line 118) Logging 
initialized
 INFO [main] 2013-08-07 14:56:46,807 CassandraDaemon.java (line 145) JVM 
vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_25
 INFO [main] 2013-08-07 14:56:46,808 CassandraDaemon.java (line 183) Heap size: 
8248098816/8248098816
 INFO [main] 2013-08-07 14:56:46,808 CassandraDaemon.java (line 184) Classpath: 
/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/conf:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/build/classes/main:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/build/classes/thrift:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/antlr-3.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/avro-1.4.0-fixes.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/avro-1.4.0-sources-fixes.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-cli-1.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-codec-1.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-lang-2.6.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/compress-lzf-0.8.4.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/concurrentlinkedhashmap-lru-1.3.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/guava-13.0.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/high-scale-lib-1.1.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jackson-core-asl-1.9.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jackson-mapper-asl-1.9.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jamm-0.2.5.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jbcrypt-0.3m.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jline-1.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/json-simple-1.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/libthrift-0.7.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/log4j-1.2.16.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/lz4-1.1.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/metrics-core-2.0.3.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/netty-3.5.9.Final.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/servlet-api-2.5-20081211.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/slf4j-api-1.7.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/slf4j-log4j12-1.7.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snakeyaml-1.6.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snappy-java-1.0.5.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snaptree-0.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jamm-0.2.5.jar
 INFO [main] 2013-08-07 14:56:46,822 CLibrary.java (line 65) JNA not found. 
Native methods will be disabled.
 INFO [main] 2013-08-07 14:56:46,891 DatabaseDescriptor.java (line 132) Loading 
settings from 
file:/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/conf/cassandra.yaml
 INFO [main] 2013-08-07 14:56:47,821 DatabaseDescriptor.java (line 150) Data 
files directories: [/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/data]
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 151) Commit 
log directory: /opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/commitlogs
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 191) 
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 205) 
disk_failure_policy is stop
 INFO [main] 2013-08-07 14:56:48,000 DatabaseDescriptor.java (line 273) Global 
memtable threshold is enabled at 2622MB
 INFO [main] 2013-08-07 14:56:49,142 DatabaseDescriptor.java (line 401) Not 
using multi-threaded compaction
 INFO [main] 2013-08-07 14:56:52,610 CacheService.java (line 111) Initializing 
key cache with capacity of 100 MBs.
 INFO [main] 2013-08-07 14:56:52,675 CacheService.java (line 140) Scheduling 
key cache save to each 14400 seconds (going to save all keys).
 INFO [main] 2013-08-07 14:56:52,678 CacheService.java (line 154) Initializing 
row cache with capacity of 0 MBs and provider 
org.apache.cassandra.cache.SerializingCacheProvider
 INFO [main] 2013-08-07 14:56:52,698 CacheService.java (line 166) Scheduling 
row cache save to each 0 seconds (going to save all keys).
 INFO [main] 2013-08-07 14:56:57,163 DatabaseDescriptor.java (line 535) 
Couldn't detect any schema definitions in local storage.
 INFO [main] 2013-08-07 14:56:57,164 DatabaseDescriptor.java (line 540) To 
create keyspaces and column families, see 'help create' in cqlsh.
 INFO [main] 2013-08-07 14:56:57,258 CommitLog.java (line 120) No commitlog 
files found; skipping replay
 INFO [main] 2013-08-07 14:56:57,715 StorageService.java (line 456) Cassandra 
version: 1.2.8-SNAPSHOT
 INFO [main] 2013-08-07 14:56:57,715 StorageService.java (line 457) Thrift API 
version: 19.36.0
 INFO [main] 2013-08-07 14:56:57,716 StorageService.java (line 458) CQL 
supported versions: 2.0.0,3.0.5 (default: 3.0.5)
 INFO [main] 2013-08-07 14:56:57,795 StorageService.java (line 483) Loading 
persisted ring state
 INFO [main] 2013-08-07 14:56:57,805 StorageService.java (line 564) Starting up 
server gossip
 WARN [main] 2013-08-07 14:56:57,823 SystemTable.java (line 573) No host ID 
found, created 6abf69c6-8472-4607-96b7-4a532ed8e2a8 (Note: This should happen 
exactly once per node).
 INFO [main] 2013-08-07 14:56:57,852 ColumnFamilyStore.java (line 630) 
Enqueuing flush of Memtable-local@2038679449(367/367 serialized/live bytes, 15 
ops)
ERROR [FlushWriter:1] 2013-08-07 14:56:57,860 CassandraDaemon.java (line 192) 
Exception in thread Thread[FlushWriter:1,5,main]
java.lang.RuntimeException: Insufficient disk space to write 452 bytes
        at 
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:42)
        at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,243 GCInspector.java (line 119) GC 
for ParNew: 2786 ms for 1 collections, 230275096 used; max is 8248098816
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,250 StatusLogger.java (line 53) 
Pool Name                    Active   Pending   Blocked
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,291 StatusLogger.java (line 68) 
MemtablePostFlusher               1         1         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,292 StatusLogger.java (line 68) 
FlushWriter                       0         0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,293 StatusLogger.java (line 68) 
commitlog_archiver                0         0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,294 StatusLogger.java (line 73) 
CompactionManager                 0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,522 StatusLogger.java (line 85) 
MessagingService                n/a       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,523 StatusLogger.java (line 95) 
Cache Type                     Size                 Capacity               
KeysToSave                                                         Provider
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,524 StatusLogger.java (line 96) 
KeyCache                          0                104857600                    
  all                                                                 
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,525 StatusLogger.java (line 102) 
RowCache                          0                        0                    
  all              org.apache.cassandra.cache.SerializingCacheProvider
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,525 StatusLogger.java (line 109) 
ColumnFamily                Memtable ops,data
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,526 StatusLogger.java (line 112) 
system.local                              0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,526 StatusLogger.java (line 112) 
system.peers                              0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,527 StatusLogger.java (line 112) 
system.batchlog                           0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,527 StatusLogger.java (line 112) 
system.NodeIdInfo                         0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,528 StatusLogger.java (line 112) 
system.LocationInfo                       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,528 StatusLogger.java (line 112) 
system.Schema                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,529 StatusLogger.java (line 112) 
system.Migrations                         0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,529 StatusLogger.java (line 112) 
system.schema_keyspaces                 8,251
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) 
system.schema_columns               398,24717
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) 
system.schema_columnfamilies           369,22187
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) 
system.IndexInfo                          0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,531 StatusLogger.java (line 112) 
system.range_xfers                        0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,531 StatusLogger.java (line 112) 
system.peer_events                        0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) 
system.hints                              0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) 
system.HintsColumnFamily                  0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) 
system_traces.sessions                    0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,533 StatusLogger.java (line 112) 
system_traces.events                      0,0

                
> Crash caused by insufficient disk space to flush
> ------------------------------------------------
>
>                 Key: CASSANDRA-5605
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5605
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.5
>         Environment: java version "1.7.0_15"
>            Reporter: Dan Hendry
>            Priority: Minor
>             Fix For: 2.0.1
>
>
> A few times now I have seen our Cassandra nodes crash by running themselves 
> out of memory. It starts with the following exception:
> {noformat}
> ERROR [FlushWriter:13000] 2013-05-31 11:32:02,350 CassandraDaemon.java (line 
> 164) Exception in thread Thread[FlushWriter:13000,5,main]
> java.lang.RuntimeException: Insufficient disk space to write 8042730 bytes
>         at 
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:42)
>         at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:722)
> {noformat} 
> After which, it seems the MemtablePostFlusher stage gets stuck and no further 
> memtables get flushed: 
> {noformat} 
> INFO [ScheduledTasks:1] 2013-05-31 11:59:12,467 StatusLogger.java (line 68) 
> MemtablePostFlusher               1        32         0
> INFO [ScheduledTasks:1] 2013-05-31 11:59:12,469 StatusLogger.java (line 73) 
> CompactionManager                 1         2
> {noformat} 
> What makes this ridiculous is that, at the time, the data directory on this 
> node had 981GB free disk space (as reported by du). We primarily use STCS and 
> at the time the aforementioned exception occurred, at least one compaction 
> task was executing which could have easily involved 981GB (or more) worth of 
> input SSTables. Correct me if I am wrong but but Cassandra counts data 
> currently being compacted against available disk space. In our case, this is 
> a significant overestimation of the space required by compaction since a 
> large portion of the data being compacted has expired or is an overwrite.
> More to the point though, Cassandra should not crash because its out of disk 
> space unless its really actually out of disk space (ie, dont consider 
> 'phantom' compaction disk usage when flushing). I have seen one of our nodes 
> die in this way before our alerts for disk space even went off.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to