[
https://issues.apache.org/jira/browse/CASSANDRA-21615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120749#comment-18120749
]
Johnny Miller commented on CASSANDRA-21615:
-------------------------------------------
Thanks - will continue with pulling in the commits.
> UCS never compacts tiny high-density SSTables streamed by repair
> ----------------------------------------------------------------
>
> Key: CASSANDRA-21615
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21615
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction/UCS
> Reporter: Hayato Shimizu
> Assignee: Johnny Miller
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Cassandra Version: 5.0.8
> This was observed on a production cluster, and it took a while to rule out
> misconfiguration before it became clear that UCS has no configuration that
> will compact these repair-streamed SSTables.
> On tables that receive frequent repair streams, UCS accumulates a very large
> number of tiny SSTables that never get compacted. The live SSTable count
> climbs into the thousands per node while total disk usage barely increases,
> because each streamed file is only hundreds of bytes to a few KB.
> We could see in the metrics the SSTable count increased only during repair
> activity and did not decrease afterwards, and pending compactions stayed near
> zero throughout. So this was definitely not a compaction backlog. UCS did not
> appear to be selecting these SSTables for compaction in the first place.
> The following was seen on a production node, via {{{}sstablemetadata{}}}:
> * a file of a few hundred bytes, local token space coverage on the order of
> 1e-11, so density (size ÷ coverage) on the order of 1e13
> * a file of a few hundred bytes, coverage on the order of 1e-10, density on
> the order of 1e12
> * a file of a few thousand bytes, coverage on the order of 1e-11, density on
> the order of 1e14
> At these densities each file was assigned to a very high level, far above the
> table's normal SSTables. Each covered such a narrow token range that it
> intersected no other SSTable in its level, so no overlap set of size >= 2
> ever formed and the threshold was never met.
> Lowering the threshold was the obvious first thing to try, but unfortunately
> it did not help. Setting scaling_parameters to 'T4, T4, N' (threshold 2 in
> the upper levels), and separately updated min_sstable_size to 0MiB, then only
> a handful of files whose ranges happened to intersect compacted, so the table
> stayed at a very large SSTable count. This makes sense since a threshold of 2
> does nothing when nothing intersects.
> overlap_inclusion_method was on its default of TRANSITIVE throughout, so
> transitive inclusion was already in effect and did not pull these files in.
> As far as we can tell that's because they are denser than the table's larger
> SSTables, so a lower-level compaction never seems to include them, though we
> have not confirmed this in the code.
> Checked JIRA before filing and didn't find this — CASSANDRA-16552 and 16638
> are the closest but they're about the anticompaction race under incremental
> repair, which is different (this is fully unrepaired, no anticompaction).
> We'd like to know whether the above explanation is correct, whether this is a
> bug or a known limitation of UCS, and whether there's a recommended way to
> handle repair-streamed SSTables like these on UCS tables in the meantime.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
