[
https://issues.apache.org/jira/browse/CASSANDRA-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231151#comment-14231151
]
Jason Brown commented on CASSANDRA-7203:
----------------------------------------
Hmm, interesting idea. However, I suspect any install base large enough is
going to throw a shitte tonne of cache (of the memcache/redis ilk) in front of
any database, and thus probably skewing the (read) distribution of traffic that
actually makes it to the database. I have not much solid evidence to back up
this assertion, but I also feel in my gut that there is not a one-to-one
between who knocks on the front door and who actually goes to a database (or to
many of many databases).
As regards the memtable flush, you'd have to have some partitioning scheme, and
flush on those bounds - hypothetically, semi-reasonable. However, I think the
gnarly work will be in the compaction code. We now have three officially
supported compaction strategies, and I wonder how complication would be added
there. Remembering what happened with incremental repair (and the special
casing of LCS vs STCS), I'd be a bit concerned the complexity creep. Perhaps
this wouldn't apply to all the strategies (I would have to think more about
DateTiered), but even that can be seen as a special casing.
At the end of the day, if you want to go down this path, sure, we can see where
it leads, and we can evaluate the results vs. the costs involved. TBH, though,
this doesn't appear to be a huge win (I think we can all agree incremental, at
best), and I think we have bigger fish to fry.
> Flush (and Compact) High Traffic Partitions Separately
> ------------------------------------------------------
>
> Key: CASSANDRA-7203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7203
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Benedict
> Labels: compaction, performance
>
> An idea possibly worth exploring is the use of streaming count-min sketches
> to collect data over the up-time of a server to estimating the velocity of
> different partitions, so that high-volume partitions can be flushed
> separately on the assumption that they will be much smaller in number, thus
> reducing write amplification by permitting compaction independently of any
> low-velocity data.
> Whilst the idea is reasonably straight forward, it seems that the biggest
> problem here will be defining any success metric. Obviously any workload
> following an exponential/zipf/extreme distribution is likely to benefit from
> such an approach, but whether or not that would translate in real terms is
> another matter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
