Christian Winther created CASSANDRA-10393: ---------------------------------------------
Summary: LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref) Key: CASSANDRA-10393 URL: https://issues.apache.org/jira/browse/CASSANDRA-10393 Project: Cassandra Issue Type: Bug Environment: v 2.2.1 (from apt) -> lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 7.8 (wheezy) Release: 7.8 Codename: wheezy -> java -version java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) Reporter: Christian Winther When trying to repair full on a table with the following schema my nodes stall and end up with spamming this I've recently changed the table from SizeTieredCompactionStrategy to LeveledCompactionStrategy. Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well When trying to full repair post compaction change, I got "out of order" errors. A few google searches later, I was told to "scrub" the keyspace - did that during the night (no problems logged, and no data lost) Now a repair just stalls and output memory leaks all over the place {code} CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true; CREATE TABLE sessions.sessions ( id text PRIMARY KEY, client_ip text, controller text, controller_action text, created timestamp, data text, expires timestamp, http_host text, modified timestamp, request_agent text, request_agent_bot boolean, request_path text, site_id int, user_id int ) WITH bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; {code} ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected -- This message was sent by Atlassian JIRA (v6.3.4#6332)