Hello to all I have cassandra cluster with 3 nodes and RF=3 writing with Quorum. Application wrote today several millions of records to specific CF. After that one of servers went wild , he eats up the disk. As i see from logs hinted handoff from 2 other servers are occuring to disk server. On this server i see that data is flushed to disk each several seconds : WARN [CompactionExecutor:249] 2014-04-14 19:17:38,633 CompactionManager.java (line 509) insufficient space to compact all requested files SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-548-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-978-Data.db') WARN [CompactionExecutor:249] 2014-04-14 19:17:58,647 CompactionManager.java (line 509) insufficient space to compact all requested files SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-978-Data.db') INFO [COMMIT-LOG-WRITER] 2014-04-14 19:18:06,232 CommitLogSegment.java (line 59) Creating new commitlog segment /opt/cassandra/commitlog/CommitLog-1397492286232.log WARN [CompactionExecutor:249] 2014-04-14 19:18:18,648 CompactionManager.java (line 509) insufficient space to compact all requested files SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db') ERROR [CompactionExecutor:249] 2014-04-14 19:18:18,649 CompactionManager.java (line 513) insufficient space to compact even the two smallest files, aborting INFO [NonPeriodicTasks:1] 2014-04-14 19:18:25,228 MeteredFlusher.java (line 62) flushing high-traffic column family CFS(Keyspace='USER_DATA', ColumnFamily='freeNumbers') INFO [NonPeriodicTasks:1] 2014-04-14 19:18:25,228 ColumnFamilyStore.java (line 1128) Enqueuing flush of Memtable-freeNumbers@1950635535(37693440/309874109 serialized/live bytes, 837632 ops) INFO [FlushWriter:22] 2014-04-14 19:18:25,229 Memtable.java (line 237) Writing Memtable-freeNumbers@1950635535(37693440/309874109 serialized/live bytes, 837632 ops) INFO [FlushWriter:22] 2014-04-14 19:18:26,871 Memtable.java (line 254) Completed flushing /opt/cassandra/data/USER_DATA/freeNumbers-g-1066-Data.db (38103944 bytes) INFO [CompactionExecutor:251] 2014-04-14 19:18:26,872 CompactionManager.java (line 542) Compacting Minor: [SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1065-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1063-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1064-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1066-Data.db')] INFO [CompactionExecutor:251] 2014-04-14 19:18:26,878 CompactionController.java (line 146) Compacting large row USER_DATA/freeNumbers:8bdf9678-6d70-11e3-85ab-80e385abf85d (151810145 bytes) incrementally
However total data of this CF is around 4.5 GB , while disk usage for this CF on this server overcomes 20GB. I have tried restart of this server , cyclic restart of all servers and no luck it continues to write data .i can not run compact also.... How can i stop that? Best regards Yulian Oifa
