[ https://issues.apache.org/jira/browse/CASSANDRA-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom Petracca reassigned CASSANDRA-14585: ---------------------------------------- Assignee: Tom Petracca > For compactions to progress when pending is greater than 10K cassandra must > be continuously restarted and compactions_in_progress purged > ---------------------------------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-14585 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14585 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction > Environment: Apache Cassandra 2.2.12 > Reporter: Zachary Schneider > Assignee: Tom Petracca > Priority: Normal > > We are experiencing the following behavior when utilizing DTCS on nodes that > join the ring and subsequently have many pending compactions. > The symptoms are the following: > nodetool compactionstats reports compactions progressing, however pending > count is never significantly reduced. > When investigating we notice the following: > compactions_in_progress is never compacted > {code:java} > find > /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/ > | wc -l > 560730 > {code} > Eventually we will end up running into to many open files. > In order for compaction to progress we must: > * truncate compactions_in_progress > * restart Cassandra > This compactions will progress rapidly 2K or so and eventually the process > will hang again. > In the debug log we see constant flushes to compactions_in_progress. > {code:java} > DEBUG [CompactionExecutor:10] 2018-07-24 07:39:29,552 > ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428 > (0%) on-heap, 354 (0%) off-heap INFO [MemtableFlushWriter:9] 2018-07-24 > 07:39:29,552 Memtable.java:352 - Writing > Memtable-compactions_in_progress@1407915569(0.263KiB serialized bytes, 12 > ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:9] 2018-07-24 > 07:39:29,553 Memtable.java:386 - Completed flushing > /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8140-big-Data.db > (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, > position=28982298) DEBUG [CompactionExecutor:16] 2018-07-24 07:39:29,595 > ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 > (0%) on-heap, 397 (0%) off-heap INFO [MemtableFlushWriter:13] 2018-07-24 > 07:39:29,595 Memtable.java:352 - Writing > Memtable-compactions_in_progress@535072625(0.298KiB serialized bytes, 13 ops, > 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:13] 2018-07-24 > 07:39:29,595 Memtable.java:386 - Completed flushing > /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8141-big-Data.db > (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, > position=29708684) DEBUG [CompactionExecutor:15] 2018-07-24 07:39:29,619 > ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428 > (0%) on-heap, 356 (0%) off-heap INFO [MemtableFlushWriter:12] 2018-07-24 > 07:39:29,619 Memtable.java:352 - Writing > Memtable-compactions_in_progress@319301851(0.265KiB serialized bytes, 12 ops, > 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:12] 2018-07-24 > 07:39:29,620 Memtable.java:386 - Completed flushing > /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8142-big-Data.db > (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, > position=30591989) DEBUG [CompactionExecutor:9] 2018-07-24 07:39:29,660 > ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 > (0%) on-heap, 396 (0%) off-heap INFO [MemtableFlushWriter:15] 2018-07-24 > 07:39:29,660 Memtable.java:352 - Writing > Memtable-compactions_in_progress@1000370375(0.297KiB serialized bytes, 13 > ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:15] 2018-07-24 > 07:39:29,660 Memtable.java:386 - Completed flushing > /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8143-big-Data.db > (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, > position=31558471) DEBUG [CompactionExecutor:14] 2018-07-24 07:39:29,688 > ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 > (0%) on-heap, 389 (0%) off-heap INFO [MemtableFlushWriter:10] 2018-07-24 > 07:39:29,689 Memtable.java:352 - Writing > Memtable-compactions_in_progress@204811984(0.290KiB serialized bytes, 13 ops, > 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:10] 2018-07-24 > 07:39:29,689 Memtable.java:386 - Completed flushing > /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8144-big-Data.db > (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, > position=32153206) > {code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org