[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214961#comment-14214961
]
Shawn Kumar commented on CASSANDRA-8248:
Alex, have you determined the issue with resident being larger, and are you
still seeing this problem? If there are any further details you can provide
(are you carrying out incremental repairs? do compactions have any affect?),
they would be much appreciated.
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>Assignee: Shawn Kumar
> Attachments: thread_dump
>
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {noformat}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {noformat}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a81d2000-7f63a8402000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8402000-7f63a8622000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8622000-7f63a8842000 r--s 08:21 9442763
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244039#comment-14244039
]
Benedict commented on CASSANDRA-8248:
-
+1
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>Assignee: Joshua McKenzie
> Attachments: 8248_v1.txt, thread_dump
>
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {noformat}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {noformat}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a81d2000-7f63a8402000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8402000-7f63a8622000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8622000-7f63a8842000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8842000-7f63a8a62000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedb
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[ https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267300#comment-14267300 ] Rahul Bhardwaj commented on CASSANDRA-8248: --- I was also facing the same issue, as whole 64 G memory was getting consumed by cassandra. Some one suggested me to apply this patch . After applying it now i am noticing below error very frequently in my system.log. Please help in analyzing it. INFO [Service Thread] 2015-01-07 10:56:26,172 GCInspector.java:142 - ParNew GC in 255ms. CMS Old Gen: 4660118640 -> 4727753888; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 95561024 -> 58883904 ERROR [NonPeriodicTasks:1] 2015-01-07 10:56:27,465 CassandraDaemon.java:153 - Exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.AssertionError: null at org.apache.cassandra.io.util.Memory.free(Memory.java:300) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71] at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_71] at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] INFO [CompactionExecutor:9] 2015-01-07 10:56:27,467 ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164 (0%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:4] 2015-01-07 10:56:27,468 Memtable.java:325 - Writing Memtable-compactions_in_progress@1148337401(0 serialized bytes, 1 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:4] 2015-01-07 10:56:27,476 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1331-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1420608014522, position=60129547) ERROR [CompactionExecutor:9] 2015-01-07 10:56:27,477 CassandraDaemon.java:153 - Exception in thread Thread[CompactionExecutor:9,1,main] java.lang.AssertionError: null at org.apache.cassandra.io.util.Memory.free(Memory.java:300) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71] at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_71] at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] INFO [CompactionExecutor:10] 2015-01-07 10:56:27,478 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/compactions_
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267989#comment-14267989
]
Joshua McKenzie commented on CASSANDRA-8248:
[~rahul.bhardwaj]: Are you seeing a large number of tmplink files accumulate on
your filesystem after compaction completes which led you to applying this
patch? As mentioned in CASSANDRA-8061, it's unlikely the posted patch on this
ticket fixes the tmplink file issue but rather addresses a potential
handle/memory leak.
We don't check for 0 size memory on deserialize in OffHeapBitset so it's
possible you have a corrupt BloomFilter that's 0 size. It'll deserialize and
read in as 0 and then fail on assertion when it comes time to close the file.
A [nodetool
scrub|http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsScrub.html]
against the problematic column family may help expose that.
In general, it's probably worth adding an assertion to OffHeapBitSet and
OpenBitSet's respective deserialize calls to check for 0 size and fail with an
informative (i.e. point to suspect file) error.
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>Assignee: Joshua McKenzie
> Attachments: 8248_v1.txt, thread_dump
>
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {noformat}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {noformat}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200472#comment-14200472
]
Michael Shuler commented on CASSANDRA-8248:
---
Repair is an expensive process. After repair has completed and the JVM has
garbage collected, what are you seeing? If things return to expected resource
use, this sounds like expected behavior.
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {quote}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a81d2000-7f63a8402000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8402000-7f63a8622000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8622000-7f63a8842000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (delete
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200544#comment-14200544
]
Alexander Sterligov commented on CASSANDRA-8248:
No, it doesn't return to expected resources after forcing GC.
Just in case:
tpstats:
{quote}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 06868466 0
0
RequestResponseStage 0 0 408799535 0
0
MutationStage 0 0 198570382 0
0
ReadRepairStage 0 0 46197 0
0
GossipStage 0 02150908 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage0 0126 0
0
ValidationExecutor0 0 72747 0
0
MemtableReclaimMemory 0 0 97987 0
0
InternalResponseStage 0 0 41583 0
0
AntiEntropyStage 0 0 630829 0
0
MiscStage 0 0 0 0
0
CommitLogArchiver 0 0 0 0
0
MemtableFlushWriter 0 0 97987 0
0
PendingRangeCalculator0 0 62 0
0
MemtablePostFlush 0 0 191444 0
0
CompactionExecutor0 04131622 0
0
AntiEntropySessions 0 0512 0
0
HintedHandoff 0 0421 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 30193
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{quote}
compactionstats:
{quote}
pending tasks: 0
Active compaction remaining time :n/a
{quote}
GC was just finished:
{quote}
INFO [Service Thread] 2014-11-06 21:25:05,233 GCInspector.java:88 -
ConcurrentMarkSweep GC in 15275ms. CMS Old Gen: 3596661592 -> 3596726792; Par
Eden Space: 1371921168 -> 0;
{quote}
It still uses ~51g of RAM.
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {quote}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandr
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200699#comment-14200699
]
Michael Shuler commented on CASSANDRA-8248:
---
Anything interesting in thread and heap dumps? If you are only looking at the
value top gives you, your initial post had 55G resident - you just posted 51G
(assume this to also be resident). I don't see where the leak is.
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {quote}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a81d2000-7f63a8402000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8402000-7f63a8622000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8622000-7f63a8842000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-k
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200707#comment-14200707
]
Michael Shuler commented on CASSANDRA-8248:
---
How much physical RAM is in the machine and what are your heap configs?
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {quote}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a81d2000-7f63a8402000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8402000-7f63a8622000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8622000-7f63a8842000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8842000-7f63a8a62000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d3
[jira] [Commented] (CASSANDRA-8248) Possible memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201867#comment-14201867
]
Alexander Sterligov commented on CASSANDRA-8248:
Host is 48GB ram. Our linux kernel specialist now investigates why res is
reported to be 51GB (main theory is that it's a rounding error). I hope he will
be able to collect more details about this problem.
Xmx is 8GB.
It looks like a lot of already removed sstable files are kept mmaped. Actually
it's not a memory leak, but mmaped inodes won't be removed from disk, so it's a
file space leak.
> Possible memory leak
> -
>
> Key: CASSANDRA-8248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports:
> {quote}
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6332000-7f63a6562000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6562000-7f63a6792000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6792000-7f63a69c2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a6e22000-7f63a7052000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7052000-7f63a7282000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7282000-7f63a74b2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a74b2000-7f63a76e2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a76e2000-7f63a7912000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7912000-7f63a7b42000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7b42000-7f63a7d72000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a81d2000-7f63a8402000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (deleted)
> 7f63a8402000-7f63a8622000 r--s 08:21 9442763
> /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
> (delet
