Here is the snapshot of the logs from one of the machine between the time when I mailed to when it finally started to look at the commitlogs. At the end you can see that it is discarding obsolete commitlogs. Not sure what that means.
INFO [HintedHandoff:1] 2011-05-17 01:17:10,510 ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable-HintsColumnFamily@866759847(153737 bytes, 3271 operations) INFO [FlushWriter:309] 2011-05-17 01:17:10,511 Memtable.java (line 158) Writing Memtable-HintsColumnFamily@866759847(153737 bytes, 3271 operations) INFO [CompactionExecutor:1] 2011-05-17 01:17:10,511 CompactionManager.java (line 395) Compacting [SSTableReader(path='/mnt/cassandra/data/system/HintsColumnFamily-f-92-Data.db'),SSTableReader(path='/mnt/cassandra/data/system/HintsColumnFamily-f-91-Data.db')] INFO [FlushWriter:309] 2011-05-17 01:17:10,652 Memtable.java (line 165) Completed flushing /mnt/cassandra/data/system/HintsColumnFamily-f-93-Data.db (375072 bytes) INFO [CompactionExecutor:1] 2011-05-17 01:17:10,654 CompactionManager.java (line 482) Compacted to /mnt/cassandra/data/system/HintsColumnFamily-tmp-f-94-Data.db. 514,078 to 511,862 (~99% of original) bytes for 2 keys. Time: 142ms. INFO [HintedHandoff:1] 2011-05-17 01:17:10,654 HintedHandOffManager.java (line 360) Finished hinted handoff of 3271 rows to endpoint /10.32.6.238 INFO [COMMIT-LOG-WRITER] 2011-05-17 01:17:10,753 CommitLog.java (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1305593058271.log) INFO [COMMIT-LOG-WRITER] 2011-05-17 01:17:10,754 CommitLog.java (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1305593776786.log) INFO [COMMIT-LOG-WRITER] 2011-05-17 01:17:10,754 CommitLog.java (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1305593808049.log) INFO [COMMIT-LOG-WRITER] 2011-05-17 01:17:10,754 CommitLog.java (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1305593840228.log) INFO [HintedHandoff:1] 2011-05-17 01:17:58,010 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.32.6.238 INFO [HintedHandoff:1] 2011-05-17 01:17:58,111 ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable-HintsColumnFamily@1025764186(47 bytes, 1 operations) INFO [FlushWriter:309] 2011-05-17 01:17:58,111 Memtable.java (line 158) Writing Memtable-HintsColumnFamily@1025764186(47 bytes, 1 operations) INFO [CompactionExecutor:1] 2011-05-17 01:17:58,112 CompactionManager.java (line 395) Compacting [SSTableReader(path='/mnt/cassandra/data/system/HintsColumnFamily-f-94-Data.db'),SSTableReader(path='/mnt/cassandra/data/system/HintsColumnFamily-f-93-Data.db')] INFO [FlushWriter:309] 2011-05-17 01:17:58,119 Memtable.java (line 165) Completed flushing /mnt/cassandra/data/system/HintsColumnFamily-f-95-Data.db (127 bytes) INFO [CompactionExecutor:1] 2011-05-17 01:17:58,192 CompactionManager.java (line 482) Compacted to /mnt/cassandra/data/system/HintsColumnFamily-tmp-f-96-Data.db. 886,934 to 123,830 (~13% of original) bytes for 2 keys. Time: 80ms. INFO [HintedHandoff:1] 2011-05-17 01:17:58,192 HintedHandOffManager.java (line 360) Finished hinted handoff of 1 rows to endpoint /10.32.6.238 INFO [NonPeriodicTasks:1] 2011-05-17 02:58:51,856 SSTable.java (line 147) Deleted /mnt/cassandra/data/system/HintsColumnFamily-f-93 INFO [NonPeriodicTasks:1] 2011-05-17 02:58:51,857 SSTable.java (line 147) Deleted /mnt/cassandra/data/system/HintsColumnFamily-f-94 INFO [CompactionExecutor:1] 2011-05-17 04:39:16,679 CacheWriter.java (line 96) Saved ObjectKeySpace-Profile-KeyCache (200000 items) in 352 ms INFO [CompactionExecutor:1] 2011-05-17 04:42:24,773 CacheWriter.java (line 96) Saved ObjectKeySpace-Location-KeyCache (200000 items) in 392 ms INFO [CompactionExecutor:1] 2011-05-17 04:43:05,117 CacheWriter.java (line 96) Saved ObjectKeySpace-SpaceLocation-KeyCache (186874 items) in 358 ms INFO [CompactionExecutor:1] 2011-05-17 04:43:35,001 CacheWriter.java (line 96) Saved ObjectKeySpace-PartitionInfo-KeyCache (29073 items) in 38 ms INFO [MutationStage:886] 2011-05-17 06:32:02,024 ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable-PartitionInfo@1500472048(136213003 bytes, 2081174 operations) INFO [FlushWriter:310] 2011-05-17 06:32:02,036 Memtable.java (line 158) Writing Memtable-PartitionInfo@1500472048(136213003 bytes, 2081174 operations) INFO [FlushWriter:310] 2011-05-17 06:32:11,001 Memtable.java (line 165) Completed flushing /mnt/cassandra/data/ObjectKeySpace/PartitionInfo-f-350-Data.db (137487577 bytes) INFO [NonPeriodicTasks:1] 2011-05-17 06:35:47,390 SSTable.java (line 147) Deleted /mnt/cassandra/data/system/HintsColumnFamily-f-92 INFO [NonPeriodicTasks:1] 2011-05-17 06:35:47,391 SSTable.java (line 147) Deleted /mnt/cassandra/data/system/HintsColumnFamily-f-91 INFO [MutationStage:899] 2011-05-17 06:41:37,170 ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable-Location@1919679451(54032744 bytes, 619317 operations) INFO [FlushWriter:311] 2011-05-17 06:41:37,253 Memtable.java (line 158) Writing Memtable-Location@1919679451(54032744 bytes, 619317 operations) INFO [MutationStage:891] 2011-05-17 06:41:41,856 ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable-SpaceLocation@1584089985(72868644 bytes, 619316 operations) INFO [FlushWriter:311] 2011-05-17 06:41:51,044 Memtable.java (line 165) Completed flushing /mnt/cassandra/data/ObjectKeySpace/Location-f-238-Data.db (89292847 bytes) INFO [FlushWriter:311] 2011-05-17 06:41:51,044 Memtable.java (line 158) Writing Memtable-SpaceLocation@1584089985(72868644 bytes, 619316 operations) INFO [COMMIT-LOG-WRITER] 2011-05-17 06:41:51,057 CommitLog.java (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1305592708312.log) INFO [FlushWriter:311] 2011-05-17 06:41:56,915 Memtable.java (line 165) Completed flushing /mnt/cassandra/data/ObjectKeySpace/SpaceLocation-f-191-Data.db (81569810 bytes) INFO [COMMIT-LOG-WRITER] 2011-05-17 06:41:56,930 CommitLog.java (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1305589784797. On Tue, May 17, 2011 at 10:49 AM, mcasandra <mohitanch...@gmail.com> wrote: > Do you see anything in log files? > > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Commitlog-Disk-Full-tp6356797p6374234.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. >
