[
https://issues.apache.org/jira/browse/CASSANDRA-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17900301#comment-17900301
]
Branimir Lambov commented on CASSANDRA-18802:
---------------------------------------------
Some peformance test data from a 1TB throughput test at 90% write and 10% read:
!image-2024-11-22-11-31-22-787.png!
(orange is baseline, green CASSANDRA-20092, blue CASSANDRA-18802, further
details in the linked perf report)
Stats at the end of the throughput stage:
Baseline
{code:java}
nodetool tablestats density
Keyspace: density
Read Count: 999999996
Read Latency: 1.126 ms
Write Count: 8999999964
Write Latency: 0.096 ms
Pending Flushes: 0
Space used (live): 1198658991147
Space used (total): 1198658991147
Table: blobs
SSTable count: 1557
Old SSTable count: 0
Max SSTable size: 3835743840
Space used (live): 1198658991147
Space used (total): 1198658991147
Space used by snapshots (total): 0
Off heap memory used (total): 15364135232
SSTable Compression Ratio: 0.942
Number of partitions (estimate): 8926835792
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 2825
Speculative retries: 0
Local read count: 999999996
Local read latency: 0.920 ms
Local write count: 8999999964
Local write latency: 0.011 ms
Local read/write ratio: 0.111
Pending flushes: 0
Percent repaired: 0.0
Bytes repaired: 0
Bytes unrepaired: 1172528505501
Bytes pending repair: 0
Bloom filter false positives: 94390770
Bloom filter false ratio: 0.010
Bloom filter space used: 14643750976
Bloom filter off heap memory used: 14627761696
Index summary off heap memory used: 0
Compression metadata off heap memory used: 736373536
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 149
Compacted partition mean bytes: 149
Average live cells per slice (last five minutes): 1.00
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.00
Maximum tombstones per slice (last five minutes): 1
Droppable tombstone ratio: 0.000
nodetool compactionstats --human-readable
concurrent compactors 30
pending tasks 96
density blobs 96
compactions completed 411
data compacted 1.28 TiB
compactions aborted 0
compactions reduced 0
sstables dropped from compaction 0
15 minute rate 0.37/minute
mean rate 46.23/hour
compaction throughput (MiB/s) throttling disabled (0)
current compaction throughput (1 minute) 96.955 MiB/s
current compaction throughput (5 minute) 90.879 MiB/s
current compaction throughput (15 minute) 72.797 MiB/s
id compaction type keyspace
table completed total unit progress
ec7ed890-a80a-11ef-a32d-e1c1789989ea Compaction density
blobs 18.76 GiB 24.68 GiB bytes 76.02%
c74ed040-a817-11ef-a32d-e1c1789989ea Compaction density
blobs 9.61 GiB 12.33 GiB bytes 77.92%
710bd940-a81b-11ef-a32d-e1c1789989ea Compaction density
blobs 7.22 GiB 12.33 GiB bytes 58.59%
78de7410-a81c-11ef-a32d-e1c1789989ea Compaction density
blobs 6.53 GiB 12.33 GiB bytes 52.98%
023c29f0-a81d-11ef-a32d-e1c1789989ea Compaction density
blobs 6.18 GiB 12.33 GiB bytes 50.08%
206a8bf0-a823-11ef-a32d-e1c1789989ea Compaction density
blobs 1.93 GiB 3.08 GiB bytes 62.72%
51fe15f0-a824-11ef-a32d-e1c1789989ea Compaction density
blobs 311.95 MiB 3.08 GiB bytes 9.88%
bacdf800-a808-11ef-a32d-e1c1789989ea Compaction density
blobs 20.21 GiB 24.68 GiB bytes 81.89%
105c5b50-a821-11ef-a32d-e1c1789989ea Compaction density
blobs 3.52 GiB 12.33 GiB bytes 28.59%
99108ac0-a821-11ef-a32d-e1c1789989ea Compaction density
blobs 3.18 GiB 12.33 GiB bytes 25.76%
5320c400-a824-11ef-a32d-e1c1789989ea Compaction density
blobs 299.17 MiB 3.08 GiB bytes 9.48%
cb094cc0-a802-11ef-a32d-e1c1789989ea Compaction density
blobs 24.63 GiB 24.68 GiB bytes 99.78%
bdd18820-a815-11ef-a32d-e1c1789989ea Compaction density
blobs 10.86 GiB 12.33 GiB bytes 88.06%
c9818530-a80b-11ef-a32d-e1c1789989ea Compaction density
blobs 18.24 GiB 24.68 GiB bytes 73.87%
7fa68fa0-a810-11ef-a32d-e1c1789989ea Compaction density
blobs 14.92 GiB 24.68 GiB bytes 60.47%
98d14890-a805-11ef-a32d-e1c1789989ea Compaction density
blobs 22.56 GiB 24.68 GiB bytes 91.40%
817779e0-a822-11ef-a32d-e1c1789989ea Compaction density
blobs 2.52 GiB 24.68 GiB bytes 10.21%
236b1930-a80c-11ef-a32d-e1c1789989ea Compaction density
blobs 17.92 GiB 24.68 GiB bytes 72.62%
e0c473e0-a817-11ef-a32d-e1c1789989ea Compaction density
blobs 9.43 GiB 12.33 GiB bytes 76.51%
fc278410-a821-11ef-a32d-e1c1789989ea Compaction density
blobs 3.03 GiB 24.68 GiB bytes 12.28%
9e815d40-a821-11ef-a32d-e1c1789989ea Compaction density
blobs 3.17 GiB 12.33 GiB bytes 25.70%
a2274f10-a806-11ef-a32d-e1c1789989ea Compaction density
blobs 21.9 GiB 24.68 GiB bytes 88.71%
ed111990-a822-11ef-a32d-e1c1789989ea Compaction density
blobs 2.15 GiB 3.08 GiB bytes 69.91%
338309a0-a815-11ef-a32d-e1c1789989ea Compaction density
blobs 11.24 GiB 12.33 GiB bytes 91.18%
820e1360-a808-11ef-a32d-e1c1789989ea Compaction density
blobs 20.59 GiB 24.68 GiB bytes 83.43%
077d3d10-a821-11ef-a32d-e1c1789989ea Compaction density
blobs 3.55 GiB 12.33 GiB bytes 28.83%
67800040-a81b-11ef-a32d-e1c1789989ea Compaction density
blobs 7.27 GiB 12.33 GiB bytes 58.97%
457a1d80-a822-11ef-a32d-e1c1789989ea Compaction density
blobs 2.77 GiB 3.08 GiB bytes 89.74%
50905820-a803-11ef-a32d-e1c1789989ea Compaction density
blobs 24.17 GiB 24.68 GiB bytes 97.92%
49152920-a823-11ef-a32d-e1c1789989ea Compaction density
blobs 1.73 GiB 3.08 GiB bytes 56.20%
Keyspace size after initial workload of 1B cycles: 1.4T .
SSTable count: 1561{code}
CASSANDRA-20092
{code:java}
nodetool tablestats density
Keyspace: density
Read Count: 999999996
Read Latency: 1.086 ms
Write Count: 8999999964
Write Latency: 0.095 ms
Pending Flushes: 0
Space used (live): 1198899880878
Space used (total): 1198899880878
Table: blobs
SSTable count: 1183
Old SSTable count: 0
Max SSTable size: 3824131179
Space used (live): 1198899880878
Space used (total): 1198899880878
Space used by snapshots (total): 0
Off heap memory used (total): 14661904240
SSTable Compression Ratio: 0.942
Number of partitions (estimate): 8926835792
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 2825
Speculative retries: 0
Local read count: 999999996
Local read latency: 0.595 ms
Local write count: 8999999964
Local write latency: 0.010 ms
Local read/write ratio: 0.111
Pending flushes: 0
Percent repaired: 0.0
Bytes repaired: 0
Bytes unrepaired: 1173014759679
Bytes pending repair: 0
Bloom filter false positives: 61211813
Bloom filter false ratio: 0.009
Bloom filter space used: 13959915128
Bloom filter off heap memory used: 13959905664
Index summary off heap memory used: 0
Compression metadata off heap memory used: 701998576
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 149
Compacted partition mean bytes: 149
Average live cells per slice (last five minutes): 1.00
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.00
Maximum tombstones per slice (last five minutes): 1
Droppable tombstone ratio: 0.000
nodetool compactionstats --human-readable
concurrent compactors 30
pending tasks 74
density blobs 74
compactions completed 428
data compacted 1.4 TiB
compactions aborted 0
compactions reduced 0
sstables dropped from compaction 0
15 minute rate 0.59/minute
mean rate 48.97/hour
compaction throughput (MiB/s) throttling disabled (0)
current compaction throughput (1 minute) 96.867 MiB/s
current compaction throughput (5 minute) 93.756 MiB/s
current compaction throughput (15 minute) 79.818 MiB/s
id compaction type keyspace
table completed total unit progress
13578800-a823-11ef-a6cc-533198867eb8 Compaction density
blobs 310.8 MiB 24.68 GiB bytes 1.23%
7bdf2860-a805-11ef-a6cc-533198867eb8 Compaction density
blobs 23.86 GiB 24.68 GiB bytes 96.68%
83f41b00-a814-11ef-a6cc-533198867eb8 Compaction density
blobs 12.73 GiB 24.68 GiB bytes 51.57%
5e83e620-a805-11ef-a6cc-533198867eb8 Compaction density
blobs 23.84 GiB 24.68 GiB bytes 96.58%
42d71530-a81b-11ef-a6cc-533198867eb8 Compaction density
blobs 7.34 GiB 12.33 GiB bytes 59.54%
7a4c5490-a81f-11ef-a6cc-533198867eb8 Compaction density
blobs 4.18 GiB 12.33 GiB bytes 33.93%
3a579560-a816-11ef-a6cc-533198867eb8 Compaction density
blobs 10.46 GiB 12.33 GiB bytes 84.88%
52e4ed20-a81c-11ef-a6cc-533198867eb8 Compaction density
blobs 7.1 GiB 24.68 GiB bytes 28.77%
a889cb30-a81f-11ef-a6cc-533198867eb8 Compaction density
blobs 4.26 GiB 24.68 GiB bytes 17.26%
28db6bf0-a80b-11ef-a6cc-533198867eb8 Compaction density
blobs 19.64 GiB 24.68 GiB bytes 79.58%
6bc38fa0-a816-11ef-a6cc-533198867eb8 Compaction density
blobs 11.36 GiB 24.68 GiB bytes 46.02%
3a554a70-a821-11ef-a6cc-533198867eb8 Compaction density
blobs 2.78 GiB 24.68 GiB bytes 11.25%
3b572b10-a816-11ef-a6cc-533198867eb8 Compaction density
blobs 11.64 GiB 24.68 GiB bytes 47.15%
554756e0-a80b-11ef-a6cc-533198867eb8 Compaction density
blobs 19.49 GiB 24.68 GiB bytes 78.95%
e58cd160-a817-11ef-a6cc-533198867eb8 Compaction density
blobs 10.35 GiB 24.68 GiB bytes 41.96%
8fec0010-a816-11ef-a6cc-533198867eb8 Compaction density
blobs 11.29 GiB 24.68 GiB bytes 45.76%
012972b0-a823-11ef-a6cc-533198867eb8 Compaction density
blobs 412.89 MiB 3.08 GiB bytes 13.08%
b8a2ae60-a81a-11ef-a6cc-533198867eb8 Compaction density
blobs 7.65 GiB 12.33 GiB bytes 62.07%
0d8ff1b0-a81d-11ef-a6cc-533198867eb8 Compaction density
blobs 6.48 GiB 24.68 GiB bytes 26.24%
0b250f60-a821-11ef-a6cc-533198867eb8 Compaction density
blobs 2.94 GiB 3.08 GiB bytes 95.54%
36e529f0-a821-11ef-a6cc-533198867eb8 Compaction density
blobs 2.74 GiB 3.08 GiB bytes 88.87%
a17a8390-a822-11ef-a6cc-533198867eb8 Compaction density
blobs 982.81 MiB 3.08 GiB bytes 31.14%
65b78d90-a821-11ef-a6cc-533198867eb8 Compaction density
blobs 2.56 GiB 24.68 GiB bytes 10.38%
ef944200-a822-11ef-a6cc-533198867eb8 Compaction density
blobs 528.91 MiB 3.08 GiB bytes 16.76%
ddfd43f0-a81f-11ef-a6cc-533198867eb8 Compaction density
blobs 4.03 GiB 24.68 GiB bytes 16.34%
2a98f870-a821-11ef-a6cc-533198867eb8 Compaction density
blobs 2.8 GiB 3.08 GiB bytes 91.02%
520c31a0-a822-11ef-a6cc-533198867eb8 Compaction density
blobs 1.43 GiB 3.08 GiB bytes 46.34%
4b9a4a30-a81f-11ef-a6cc-533198867eb8 Compaction density
blobs 4.36 GiB 12.33 GiB bytes 35.39%
76a63b80-a815-11ef-a6cc-533198867eb8 Compaction density
blobs 11.03 GiB 12.33 GiB bytes 89.48%
622c6b00-a821-11ef-a6cc-533198867eb8 Compaction density
blobs 2.58 GiB 3.08 GiB bytes 83.77%
active compaction remaining time n/a
Keyspace size after initial workload of 1B cycles: 1.4T .
SSTable count: 1183
{code}
CASSANDRA-18802
{code:java}
nodetool tablestats density
Keyspace: density
Read Count: 999999996
Read Latency: 0.883 ms
Write Count: 8999999964
Write Latency: 0.096 ms
Pending Flushes: 0
Space used (live): 1207795635119
Space used (total): 1207795635119
Table: blobs
SSTable count: 596
Old SSTable count: 0
Max SSTable size: 4345016986
Space used (live): 1207795635119
Space used (total): 1207795635119
Space used by snapshots (total): 0
Off heap memory used (total): 13206827920
SSTable Compression Ratio: 0.942
Number of partitions (estimate): 8926835792
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 2826
Speculative retries: 0
Local read count: 999999996
Local read latency: 0.266 ms
Local write count: 8999999964
Local write latency: 0.011 ms
Local read/write ratio: 0.111
Pending flushes: 0
Percent repaired: 0.0
Bytes repaired: 0
Bytes unrepaired: 1175047469169
Bytes pending repair: 0
Bloom filter false positives: 156122057
Bloom filter false ratio: 0.009
Bloom filter space used: 12633076672
Bloom filter off heap memory used: 12633071904
Index summary off heap memory used: 0
Compression metadata off heap memory used: 573756016
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 149
Compacted partition mean bytes: 149
Average live cells per slice (last five minutes): 1.00
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.00
Maximum tombstones per slice (last five minutes): 1
Droppable tombstone ratio: 0.000
nodetool compactionstats --human-readable
concurrent compactors 30
pending tasks 120
density blobs 120
compactions completed 1035
data compacted 1.65 TiB
compactions aborted 0
compactions reduced 0
sstables dropped from compaction 0
15 minute rate 1.48/minute
mean rate 119.05/hour
compaction throughput (MiB/s) throttling disabled (0)
current compaction throughput (1 minute) 102.481 MiB/s
current compaction throughput (5 minute) 104.308 MiB/s
current compaction throughput (15 minute) 102.096 MiB/s
id compaction type keyspace
table completed total unit progress
3a688450-a821-11ef-8001-7f4f0da0b40a Compaction density
blobs 2.34 GiB 6.22 GiB bytes 37.69%
48352630-a81f-11ef-8001-7f4f0da0b40a Compaction density
blobs 5.34 GiB 6.84 GiB bytes 78.11%
ba59baf0-a81f-11ef-8001-7f4f0da0b40a Compaction density
blobs 4.67 GiB 6.84 GiB bytes 68.31%
3a688450-a821-11ef-8002-7f4f0da0b40a Compaction density
blobs 2.14 GiB 6.22 GiB bytes 34.31%
9d0a9130-a820-11ef-8005-7f4f0da0b40a Compaction density
blobs 3.03 GiB 3.09 GiB bytes 98.17%
3a688450-a821-11ef-8004-7f4f0da0b40a Compaction density
blobs 2.33 GiB 6.22 GiB bytes 37.48%
48352630-a81f-11ef-8004-7f4f0da0b40a Compaction density
blobs 5.36 GiB 6.84 GiB bytes 78.35%
ba59baf0-a81f-11ef-8002-7f4f0da0b40a Compaction density
blobs 3.93 GiB 6.84 GiB bytes 57.45%
94ef0680-a81f-11ef-8001-7f4f0da0b40a Compaction density
blobs 4.9 GiB 6.84 GiB bytes 71.75%
7d854d90-a821-11ef-8004-7f4f0da0b40a Compaction density
blobs 1.94 GiB 6.22 GiB bytes 31.18%
94ef0680-a81f-11ef-8004-7f4f0da0b40a Compaction density
blobs 4.9 GiB 6.83 GiB bytes 71.74%
ba59baf0-a81f-11ef-8004-7f4f0da0b40a Compaction density
blobs 4.69 GiB 6.83 GiB bytes 68.63%
7d854d90-a821-11ef-8002-7f4f0da0b40a Compaction density
blobs 251.95 MiB 6.22 GiB bytes 3.95%
48352630-a81f-11ef-8002-7f4f0da0b40a Compaction density
blobs 5.32 GiB 6.84 GiB bytes 77.87%
9ba019d0-a822-11ef-8008-7f4f0da0b40a Compaction density
blobs 195.31 MiB 3.08 GiB bytes 6.18%
9ba019d0-a822-11ef-8001-7f4f0da0b40a Compaction density
blobs 193.33 MiB 3.09 GiB bytes 6.12%
9d0a9130-a820-11ef-8004-7f4f0da0b40a Compaction density
blobs 3.03 GiB 3.08 GiB bytes 98.19%
7d854d90-a821-11ef-8003-7f4f0da0b40a Compaction density
blobs 244.82 MiB 6.23 GiB bytes 3.84%
94ef0680-a81f-11ef-8003-7f4f0da0b40a Compaction density
blobs 4.89 GiB 6.84 GiB bytes 71.55%
9d0a9130-a820-11ef-8006-7f4f0da0b40a Compaction density
blobs 2.45 GiB 3.09 GiB bytes 79.47%
48352630-a81f-11ef-8003-7f4f0da0b40a Compaction density
blobs 5.31 GiB 6.84 GiB bytes 77.70%
9d0a9130-a820-11ef-8007-7f4f0da0b40a Compaction density
blobs 2.44 GiB 3.09 GiB bytes 78.92%
23350fe0-a81e-11ef-8003-7f4f0da0b40a Compaction density
blobs 6.64 GiB 6.84 GiB bytes 97.14%
9d0a9130-a820-11ef-8002-7f4f0da0b40a Compaction density
blobs 3.07 GiB 3.09 GiB bytes 99.60%
3a688450-a821-11ef-8003-7f4f0da0b40a Compaction density
blobs 2.12 GiB 6.22 GiB bytes 34.00%
ba59baf0-a81f-11ef-8003-7f4f0da0b40a Compaction density
blobs 3.92 GiB 6.83 GiB bytes 57.40%
7d854d90-a821-11ef-8001-7f4f0da0b40a Compaction density
blobs 1.92 GiB 6.22 GiB bytes 30.84%
9d0a9130-a820-11ef-8003-7f4f0da0b40a Compaction density
blobs 3.08 GiB 3.09 GiB bytes 99.73%
94ef0680-a81f-11ef-8002-7f4f0da0b40a Compaction density
blobs 4.89 GiB 6.84 GiB bytes 71.57%
9ba019d0-a822-11ef-8002-7f4f0da0b40a Compaction density
blobs 191.65 MiB 3.09 GiB bytes 6.07%
active compaction remaining time n/a
Keyspace size after initial workload of 1B cycles: 1.3T .
SSTable count: 631 {code}
> Extend compaction interfaces to provide split points at operation start
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-18802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18802
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction
> Reporter: Branimir Lambov
> Assignee: Branimir Lambov
> Priority: Normal
> Attachments: CASSANDRA-18802-perf-report.html,
> image-2024-11-22-11-31-22-787.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The current compaction interfaces allow a compaction strategy to split at
> arbitrary points while it is writing output. In some cases (e.g. UCS) we know
> in advance where we want to split. Giving this information before the
> operation starts allows it to operate on multiple segments of the output in
> parallel, i.e. parallelize within an operation rather than between
> operations, which can reduce individual operations' duration and
> significantly improve the DB's chances of keeping up with load, especially on
> L0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
