With the exception of a few little warnings on start-up about the Memtable live
ratio, there is nothing at WARN or above in the logs. Just before the JVM
terminates, there are about 10,000 threads in Reducer executor pools that look
like this in JConsole …
Name: CompactionReducer:1
State: TIMED_WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@72938aea
Total blocked: 0 Total waited: 1
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
The results from tpstats don't look too interesting…
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 0 0 3455159 0
0
RequestResponseStage 0 0 10133276 0
0
MutationStage 0 0 5898833 0
0
ReadRepairStage 0 0 2078449 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 236388 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage 0 0 0 0
0
MemtablePostFlusher 0 0 231 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 231 0
0
MiscStage 0 0 0 0
0
InternalResponseStage 0 0 0 0
0
HintedHandoff 0 0 35 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
REQUEST_RESPONSE 0
The results from info seem unremarkable as well…
Token : 153127065000000000000000000000000000000
Gossip active : true
Load : 5.6 GB
Generation No : 1325995515
Uptime (seconds) : 67199
Heap Memory (MB) : 970.32 / 1968.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
I'm using LeveledCompactionStrategy with no throttling, and I'm not changing
the default on the number of concurrent compactors.
What is interesting to me here is that Cassandra creates an executor for every
single compaction in ParallelCompactionIterable. Why couldn't we just create a
pool with Runtime.availableProcessors() Threads and be done with it?
Let me know if I left any info out.
Thanks!
Caleb Rackliffe | Software Developer
M 949.981.0159 | [email protected]
From: aaron morton <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Sun, 8 Jan 2012 16:51:50 -0500
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Lots and Lots of CompactionReducer Threads
How many threads ? Any errors in the server logs ?
What does noodtool tpstats and nodetool compactionstats say ?
Did you change compaction_strategy for the CF's ?
By default cassandra will use as many compaction threads as you have cores, see
concurrent_compactors in cassandra.yaml
Have you set the JVM heap settings ? What does nodetool info show ?
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:
Hi Everybody,
JConsole tells me I've got CompactionReducer threads stacking up, consuming
memory, and never going away. Eventually, my Java process fails because it
can't allocate any more native threads. Here's my setup…
Cassandra 1.0.5 on CentOS 6.0
4 GB of RAM
50 GB SSD HD
Memtable flush threshold = 128 MB
compaction throughput limit = 16 MB/sec
Multithreaded compaction = true
It may very well be that I'm doing something strange here, but it seems like
those compaction threads should go away eventually. I'm hoping the combination
of a low Memtable flush threshold, low compaction T/P limit, and heavy write
load doesn't mean those threads are hanging around because they're actually not
done doing their compaction tasks.
Thanks,
Caleb Rackliffe | Software Developer
M 949.981.0159 | [email protected]<mailto:[email protected]>
