Cyril Scetbon created CASSANDRA-7249:
----------------------------------------
Summary: Too many threads associated with parallel compaction
Key: CASSANDRA-7249
URL: https://issues.apache.org/jira/browse/CASSANDRA-7249
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 12.04.3 LTS
24 CPUs (hyper threading enabled)
Reporter: Cyril Scetbon
We have a lot of threads on some nodes as you can see :
node001: 560
node002: 529
node003: 4350
node004: 552
node005: 547
node006: 554
node007: 572
node008: 1444 <==
node009: 540
node010: 13691 <==
node011: 577
node012: 536
node013: 448
node014: 10295 <==
node015: 452
node016: 576
When I check what are those threads I see a lot of "Deserializer sstables".
Enabling DEBUG mode shows that a lot of actions are about parallel compaction.
What is really surprising is that it tries to deserialize a huge number of
times each sstable even if we only have 8 files for the concerned column family
:
512690 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-616-Data.db
296623 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-637-Data.db
311904 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-642-Data.db
127061 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-643-Data.db
126921 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-644-Data.db
129815 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-645-Data.db
127862 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-646-Data.db
317069 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-647-Data.db
so, in a minute Cassandra execute 2 millions of times the following code :
{code}
else
{
logger.debug("parallel eager deserialize from " + iter.getPath());
queue.put(new RowContainer(new Row(iter.getKey(),
iter.getColumnFamilyWithColumns(ArrayBackedSortedColumns.factory()))));
}
{code}
It seems to be related to
[CASSANDRA-5720|https://issues.apache.org/jira/browse/CASSANDRA-5720] cause we
got the same error on the concerned column families before the number of
threads raise. Upgrading to 2.0 is not a solution for now :(
--
This message was sent by Atlassian JIRA
(v6.2#6252)
