[
https://issues.apache.org/jira/browse/CASSANDRA-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114507#comment-18114507
]
Mykyta Bozhenko commented on CASSANDRA-19569:
---------------------------------------------
Patch: https://github.com/apache/cassandra/pull/5140
maybeReopenEarly is called for every appended partition and is meant to skip
the page cache of the source readers once per preemptive open interval. The
offline branch never updates currentlyOpenedEarlyAt, so once the output passed
the interval, the condition held for every partition: an index lookup plus
trySkipCache over [0, current position) on every source file. The online branch
already records the position; the patch does the same for the offline one.
Measured on the offline rewrite path (ext4, 50 MiB interval, JDK 11, median of
three runs): 124 MB of ~1 KB partitions goes from 6980 ms (17.0 MB/s) to 1894
ms (62.6 MB/s). The cost is per partition rather than per byte, so smaller
partitions hurt more: with ~84 B partitions it is 5.8 MB/s before and 18.0 MB/s
after.
Attribution with bpftrace on the 124 MB case: 69,394 fadvise64 calls before, 3
after; 111,048 openat calls before, 35,251 after, since trySkipCache(String,
long, long) opens and closes the data file on every call. Only 253 ms of the
roughly 5 s difference is inside fadvise64, so most of the cost is the
per-partition index lookup and file reopen around it.
Two things from the description I could not reproduce: the rewrite is equally
slow on tmpfs here (6281 ms before, 1430 ms after), so I do not see the ramdisk
asymmetry, and the absolute figure below 0.5 MB/s is well under what this
harness shows for unpatched code. Both are consistent with the reported table
simply having far smaller partitions and larger files than my test data.
Looking for a reviewer.
> sstableupgrade is very slow
> ---------------------------
>
> Key: CASSANDRA-19569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19569
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: Norbert Schultz
> Assignee: Mykyta Bozhenko
> Priority: Normal
> Fix For: 4.1.x
>
> Attachments: flamegraph_ok.png, flamegraph_sstableupgrade.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are in the process of migrating cassandra from 3.11.x to 4.1.4 and
> upgrading the sstables using sstableupgrade from Cassandra V4.1.4, from `me-`
> to `nb-` Format
> Unfortunately, the process is very very slow (less than 0.5 MB/s).
> Some observations:
> - The process is only slow on (fast) SSDs, but not on ram disks.
> - The sstables consist of many partitions (this may be unrelated)
> - The upgrade process is fast, if we use `automatic_sstable_upgrade` instead
> of the sstableupgradetool.
> - We give enough RAM (export MAX_HEAP_SIZE=8g)
> On profiling, we found out, that sstableupgrade is burning most CPU time on
> {{posix_fadvise}} (see flamegraph_sstableupgrade.png ).
> My naive interpretation of the whole {{maybeReopenEarly}} to
> {{posix_fadvise}} chain is, that the process just informs the linux kernel,
> that the written data should not be cached. If we comment out the call to
> {{NativeLibrary.trySkipCache}}, the conversion is running at expected 10MB/s
> (see flamegraph_ok.png )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
