[
https://issues.apache.org/jira/browse/CASSANDRA-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Petracca reassigned CASSANDRA-14585:
----------------------------------------
Assignee: Tom Petracca
> For compactions to progress when pending is greater than 10K cassandra must
> be continuously restarted and compactions_in_progress purged
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-14585
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14585
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Environment: Apache Cassandra 2.2.12
> Reporter: Zachary Schneider
> Assignee: Tom Petracca
> Priority: Normal
>
> We are experiencing the following behavior when utilizing DTCS on nodes that
> join the ring and subsequently have many pending compactions.
> The symptoms are the following:
> nodetool compactionstats reports compactions progressing, however pending
> count is never significantly reduced.
> When investigating we notice the following:
> compactions_in_progress is never compacted
> {code:java}
> find
> /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/
> | wc -l
> 560730
> {code}
> Eventually we will end up running into to many open files.
> In order for compaction to progress we must:
> * truncate compactions_in_progress
> * restart Cassandra
> This compactions will progress rapidly 2K or so and eventually the process
> will hang again.
> In the debug log we see constant flushes to compactions_in_progress.
> {code:java}
> DEBUG [CompactionExecutor:10] 2018-07-24 07:39:29,552
> ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428
> (0%) on-heap, 354 (0%) off-heap INFO [MemtableFlushWriter:9] 2018-07-24
> 07:39:29,552 Memtable.java:352 - Writing
> Memtable-compactions_in_progress@1407915569(0.263KiB serialized bytes, 12
> ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:9] 2018-07-24
> 07:39:29,553 Memtable.java:386 - Completed flushing
> /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8140-big-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
> position=28982298) DEBUG [CompactionExecutor:16] 2018-07-24 07:39:29,595
> ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460
> (0%) on-heap, 397 (0%) off-heap INFO [MemtableFlushWriter:13] 2018-07-24
> 07:39:29,595 Memtable.java:352 - Writing
> Memtable-compactions_in_progress@535072625(0.298KiB serialized bytes, 13 ops,
> 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:13] 2018-07-24
> 07:39:29,595 Memtable.java:386 - Completed flushing
> /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8141-big-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
> position=29708684) DEBUG [CompactionExecutor:15] 2018-07-24 07:39:29,619
> ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428
> (0%) on-heap, 356 (0%) off-heap INFO [MemtableFlushWriter:12] 2018-07-24
> 07:39:29,619 Memtable.java:352 - Writing
> Memtable-compactions_in_progress@319301851(0.265KiB serialized bytes, 12 ops,
> 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:12] 2018-07-24
> 07:39:29,620 Memtable.java:386 - Completed flushing
> /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8142-big-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
> position=30591989) DEBUG [CompactionExecutor:9] 2018-07-24 07:39:29,660
> ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460
> (0%) on-heap, 396 (0%) off-heap INFO [MemtableFlushWriter:15] 2018-07-24
> 07:39:29,660 Memtable.java:352 - Writing
> Memtable-compactions_in_progress@1000370375(0.297KiB serialized bytes, 13
> ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:15] 2018-07-24
> 07:39:29,660 Memtable.java:386 - Completed flushing
> /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8143-big-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
> position=31558471) DEBUG [CompactionExecutor:14] 2018-07-24 07:39:29,688
> ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460
> (0%) on-heap, 389 (0%) off-heap INFO [MemtableFlushWriter:10] 2018-07-24
> 07:39:29,689 Memtable.java:352 - Writing
> Memtable-compactions_in_progress@204811984(0.290KiB serialized bytes, 13 ops,
> 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:10] 2018-07-24
> 07:39:29,689 Memtable.java:386 - Completed flushing
> /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8144-big-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
> position=32153206)
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
