[
https://issues.apache.org/jira/browse/CASSANDRA-21615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120305#comment-18120305
]
Johnny Miller commented on CASSANDRA-21615:
-------------------------------------------
[~mck] [~blambov] [~brandon.williams] Just looking at latest comments on the PR
([https://github.com/apache/cassandra/pull/5213#pullrequestreview-5335820160)]
. It would be helpful to know why my original patch where we just did it by
size is not appropraite here as this fix from the datastax commits is more
invasive and would be good to establish clearly why the the size approach is
not appropriate. I am happy to continue with the broader changes needed to make
UCS actually work in production but it would be helpful to establish clearly
why size doesnt matter ;)
> UCS never compacts tiny high-density SSTables streamed by repair
> ----------------------------------------------------------------
>
> Key: CASSANDRA-21615
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21615
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction/UCS
> Reporter: Hayato Shimizu
> Assignee: Johnny Miller
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Cassandra Version: 5.0.8
> This was observed on a production cluster, and it took a while to rule out
> misconfiguration before it became clear that UCS has no configuration that
> will compact these repair-streamed SSTables.
> On tables that receive frequent repair streams, UCS accumulates a very large
> number of tiny SSTables that never get compacted. The live SSTable count
> climbs into the thousands per node while total disk usage barely increases,
> because each streamed file is only hundreds of bytes to a few KB.
> We could see in the metrics the SSTable count increased only during repair
> activity and did not decrease afterwards, and pending compactions stayed near
> zero throughout. So this was definitely not a compaction backlog. UCS did not
> appear to be selecting these SSTables for compaction in the first place.
> The following was seen on a production node, via {{{}sstablemetadata{}}}:
> * a file of a few hundred bytes, local token space coverage on the order of
> 1e-11, so density (size ÷ coverage) on the order of 1e13
> * a file of a few hundred bytes, coverage on the order of 1e-10, density on
> the order of 1e12
> * a file of a few thousand bytes, coverage on the order of 1e-11, density on
> the order of 1e14
> At these densities each file was assigned to a very high level, far above the
> table's normal SSTables. Each covered such a narrow token range that it
> intersected no other SSTable in its level, so no overlap set of size >= 2
> ever formed and the threshold was never met.
> Lowering the threshold was the obvious first thing to try, but unfortunately
> it did not help. Setting scaling_parameters to 'T4, T4, N' (threshold 2 in
> the upper levels), and separately updated min_sstable_size to 0MiB, then only
> a handful of files whose ranges happened to intersect compacted, so the table
> stayed at a very large SSTable count. This makes sense since a threshold of 2
> does nothing when nothing intersects.
> overlap_inclusion_method was on its default of TRANSITIVE throughout, so
> transitive inclusion was already in effect and did not pull these files in.
> As far as we can tell that's because they are denser than the table's larger
> SSTables, so a lower-level compaction never seems to include them, though we
> have not confirmed this in the code.
> Checked JIRA before filing and didn't find this — CASSANDRA-16552 and 16638
> are the closest but they're about the anticompaction race under incremental
> repair, which is different (this is fully unrepaired, no anticompaction).
> We'd like to know whether the above explanation is correct, whether this is a
> bug or a known limitation of UCS, and whether there's a recommended way to
> handle repair-streamed SSTables like these on UCS tables in the meantime.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
