[ https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267989#comment-14267989 ]
Joshua McKenzie commented on CASSANDRA-8248: -------------------------------------------- [~rahul.bhardwaj]: Are you seeing a large number of tmplink files accumulate on your filesystem after compaction completes which led you to applying this patch? As mentioned in CASSANDRA-8061, it's unlikely the posted patch on this ticket fixes the tmplink file issue but rather addresses a potential handle/memory leak. We don't check for 0 size memory on deserialize in OffHeapBitset so it's possible you have a corrupt BloomFilter that's 0 size. It'll deserialize and read in as 0 and then fail on assertion when it comes time to close the file. A [nodetool scrub|http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsScrub.html] against the problematic column family may help expose that. In general, it's probably worth adding an assertion to OffHeapBitSet and OpenBitSet's respective deserialize calls to check for 0 size and fail with an informative (i.e. point to suspect file) error. > Possible memory leak > --------------------- > > Key: CASSANDRA-8248 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8248 > Project: Cassandra > Issue Type: Bug > Reporter: Alexander Sterligov > Assignee: Joshua McKenzie > Attachments: 8248_v1.txt, thread_dump > > > Sometimes during repair cassandra starts to consume more memory than expected. > Total amount of data on node is about 20GB. > Size of the data directory is 66GC because of snapshots. > Top reports: > {noformat} > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 15724 loadbase 20 0 493g 55g 44g S 28 44.2 4043:24 java > {noformat} > At the /proc/15724/maps there are a lot of deleted file maps > {quote} > 7f63a6102000-7f63a6332000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a6332000-7f63a6562000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a6562000-7f63a6792000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a6792000-7f63a69c2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a69c2000-7f63a6bf2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a6bf2000-7f63a6e22000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a6e22000-7f63a7052000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a7052000-7f63a7282000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a7282000-7f63a74b2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a74b2000-7f63a76e2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a76e2000-7f63a7912000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a7912000-7f63a7b42000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a7b42000-7f63a7d72000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a7d72000-7f63a7fa2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a7fa2000-7f63a81d2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a81d2000-7f63a8402000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a8402000-7f63a8622000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a8622000-7f63a8842000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a8842000-7f63a8a62000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a8a62000-7f63a8c82000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a8c82000-7f63a8ea2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a8ea2000-7f63a90c2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > 7f63a90c2000-7f63a92e2000 r--s 00000000 08:21 9442763 > /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db > (deleted) > {quote} > {quote} > $ sudo grep deleted /proc/15724/maps | wc -l > 640118 > {quote} > {quote} > $ sudo grep -v deleted /proc/15724/maps | wc -l > 303340 > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)