[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011068#comment-13011068
]
Aaron Morton commented on CASSANDRA-2156:
-----------------------------------------
Bunch of questions again as I'm trying to understand some more of whats going
on.
# if compaction_throughput_kb_per_sec is always going be megabytes should it
change to MB
# Not in your changes but CompactionIterator.close() will stop closing files
after the first one fails.
# I'm guessing most of the time the actual and target throughput will not
match. How about moving the INFO message in throttle() to the DEBUG level? Or
only logging at INFO is the thread will sleep?
# Should there be a config setting to turn throttling on and off? Could setting
compaction_throughput_kb_per_sec to 0 disable it ?
# For my understanding: Is there a case for making the sampling interval in
CompactionIterator.getReduce() configurable? Would we want different settings
for fewer big rows vs many small rows. e.g. two CFs where one is a secondary
index for rows in the other, could be millions of cols in one an a few in
another.
I dont understand the approach to deciding what value
compaction_throughput_kb_per_sec should have. Can you add some more info and
clarify if you are talking about the per CF buckets creating during Compaction?
Final question. Would it be better to have fewer parallel compactions where
each compaction completes quickly, than more parallel compactions that take
longer to complete. Assuming that once compaction has finished read performance
and disk usage may improve. If so would limiting compaction by sizing the
compaction thread pool be effective? (I guess the down side may be starvation
for some CF's)
> Compaction Throttling
> ---------------------
>
> Key: CASSANDRA-2156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2156
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Stu Hood
> Fix For: 0.8
>
> Attachments:
> 0001-Throttle-total-compaction-to-a-configurable-throughput.txt,
> for-0.6-0001-Throttle-compaction-to-a-fixed-throughput.txt,
> for-0.6-0002-Make-compaction-throttling-configurable.txt
>
>
> Compaction is currently relatively bursty: we compact as fast as we can, and
> then we wait for the next compaction to be possible ("hurry up and wait").
> Instead, to properly amortize compaction, you'd like to compact exactly as
> fast as you need to to keep the sstable count under control.
> For every new level of compaction, you need to increase the rate that you
> compact at: a rule of thumb that we're testing on our clusters is to
> determine the maximum number of buckets a node can support (aka, if the 15th
> bucket holds 750 GB, we're not going to have more than 15 buckets), and then
> multiply the flush throughput by the number of buckets to get a minimum
> compaction throughput to maintain your sstable count.
> Full explanation: for a min compaction threshold of {{T}}, the bucket at
> level {{N}} can contain {{SsubN = T^N}} 'units' (unit == memtable's worth of
> data on disk). Every time a new unit is added, it has a {{1/SsubN}} chance of
> causing the bucket at level N to fill. If the bucket at level N fills, it
> causes {{SsubN}} units to be compacted. So, for each active level in your
> system you have {{SubN * 1 / SsubN}}, or {{1}} amortized unit to compact any
> time a new unit is added.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
