[
https://issues.apache.org/jira/browse/CASSANDRA-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lohfink updated CASSANDRA-21550:
--------------------------------------
Description:
# Zero-copy SSTable splitting for anticompaction and range streaming
## Motivation
Anticompaction and range streaming both spend most of their cost repeating work
whose result already exists on disk.
Anticompaction rewrites every row of an SSTable through multiple writers only
to place each partition into a repaired, transient, or unrepaired child.
When streaming a subset of an SSTable's ranges, the sender can transmit whole
compression chunks verbatim, but the receiver still decompresses, deserializes,
reserializes, and recompresses every row. It then rebuilds the index, filter,
summary, and metadata even though these components can be derived without
rewriting the row data.
This work is avoidable when the required bytes form contiguous runs of the
parent's compression chunks.
## Proposed approach
Introduce reusable zero-copy SSTable-splitting machinery that:
- Selects contiguous runs of compression chunks for each child SSTable.
- Retains the compressed bytes verbatim without deserializing or recompressing
rows.
- Uses byte-range reflinks to share filesystem extents when supported.
- Falls back to copying the compressed bytes when reflinks are unavailable.
- Rebuilds the index, filter, summary, statistics, and other components from
index-only passes.
- Represents compression-chunk prefixes that do not contain indexed partitions,
allowing boundary chunks to be retained safely.
- Preserves the validation, corruption detection, lifecycle, disk-space, and
compatibility guarantees expected by existing SSTable operations.
The same machinery can then be used by anticompaction and range streaming
instead of their existing row-by-row rewrite paths.
## Delivery plan
The work will be delivered incrementally so the on-disk representation and
splitting primitives can be validated before they are introduced into critical
online paths.
### Phase 1: BIG-format splitting
Add zero-copy splitting for compressed BIG-format SSTables and expose it
through an opt-in `sstablesplit --zero-copy` mode.
Using the offline splitter as the first caller provides a bounded way to
validate chunk selection, component rebuilding, extent sharing, compatibility,
verification, scrub, streaming, loading, and upgrade behavior.
This phase is limited to BIG SSTables without SSTable-attached indexes.
### Phase 2: BTI-format splitting
Add the equivalent zero-copy splitting implementation for BTI SSTables,
including the format-specific index rebuilding required by BTI.
### Phase 3: Secondary-index support
Add support for splitting SSTables that have Storage-Attached Indexes (SAI) or
legacy secondary indexes (2i). This phase must ensure that index components are
rebuilt, shared, or invalidated correctly for every generated child.
### Phase 4: Zero-copy anticompaction
Integrate the splitting machinery with anticompaction so repaired, transient,
and unrepaired children can retain existing compressed chunks instead of
rewriting every row.
The optimized anticompaction path will initially be disabled by default and
retain the existing rewrite path as a fallback.
### Phase 5: Zero-copy range streaming
Integrate the machinery with range streaming so eligible streams transfer
compressed chunk runs together with enough derived information for the receiver
to construct the resulting SSTables without deserializing and rewriting rows.
The optimized streaming path will initially be disabled by default and retain
the existing row-based path for compatibility and fallback.
## Compatibility and fallback
The new representation and protocol behavior must be explicitly version-gated.
Nodes must not create or transmit split-prefix SSTables to peers that cannot
interpret them.
Zero-copy operation must remain an optimization rather than a correctness
requirement. Unsupported formats, components, indexes, filesystems, peer
versions, or split layouts must either be rejected before modification or use
an existing safe fallback path.
Reflink support is optional. Filesystems without extent sharing still benefit
from avoiding row deserialization and recompression by copying the existing
compressed bytes.
## Expected outcome
For eligible SSTables, anticompaction and range streaming will avoid row-level
deserialization, serialization, and recompression. The remaining work will
primarily be compression-chunk transfer or extent sharing plus rebuilding
derived components from indexes.
This should substantially reduce CPU use, write amplification, temporary disk
requirements, and elapsed time for both operations while preserving the
existing behavior as a fallback.
was:
## Context
This is phase 1 of
[CASSANDRA-21550](https://issues.apache.org/jira/browse/CASSANDRA-21550).
Anticompaction and range streaming currently redo work whose result already
exists on disk. They deserialize, serialize, and recompress rows even when the
required data is a contiguous run of the parent SSTable’s compression chunks.
CASSANDRA-21550 introduces machinery to retain those compressed chunks
verbatim—using extent sharing where available—and rebuild the remaining
components from index-only passes.
## Phase 1 scope
This PR deliberately limits the first phase to compressed BIG-format SSTables
and exposes the machinery through the opt-in:
```text
sstablesplit --zero-copy
```
Starting with `sstablesplit` provides a bounded, offline caller for validating
the new on-disk representation and splitting machinery before integrating it
into anticompaction and streaming.
This phase:
- Supports compressed BIG `pa`-or-later SSTables and writes `pb` children.
- Copies existing compression chunks without deserializing or recompressing
rows.
- Uses byte-range reflinks on supported Linux filesystems to share extents.
- Falls back to copying compressed bytes when reflinks are unavailable.
- Rebuilds indexes, summaries, filters, statistics, digests, and other child
components.
- Persists the first indexed partition position so readers can safely skip
retained compression-chunk prefixes.
- Updates loading, scanning, streaming, scrub, verification, and upgrade paths
to understand split-prefix SSTables.
- Requires `storage_compatibility_mode: NONE`.
- Rejects uncompressed SSTables, unsupported components, and SSTable-attached
indexes.
The requested split size remains a maximum unless one partition’s
compression-chunk span exceeds it. Compression chunks crossing a split boundary
may be retained by both adjacent children.
## Follow-up phases
The intended sequence for CASSANDRA-21550 is:
1. **BIG zero-copy splitting** — this PR.
2. Add the equivalent splitting implementation for **BTI**.
3. Add support for SSTables with **SAI and legacy secondary indexes (2i)**.
4. Use the machinery for **zero-copy anticompaction**, initially disabled by
default.
5. Use it to make **range streaming zero-copy**, also initially disabled by
default.
Anticompaction and general range streaming integration are intentionally
outside first PR.
> Zero-copy sstable splitting for anticompaction, and partial-sstable zero-copy
> streaming
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21550
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21550
> Project: Apache Cassandra
> Issue Type: New Feature
> Reporter: Chris Lohfink
> Priority: Normal
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> # Zero-copy SSTable splitting for anticompaction and range streaming
> ## Motivation
> Anticompaction and range streaming both spend most of their cost repeating
> work whose result already exists on disk.
> Anticompaction rewrites every row of an SSTable through multiple writers only
> to place each partition into a repaired, transient, or unrepaired child.
> When streaming a subset of an SSTable's ranges, the sender can transmit whole
> compression chunks verbatim, but the receiver still decompresses,
> deserializes, reserializes, and recompresses every row. It then rebuilds the
> index, filter, summary, and metadata even though these components can be
> derived without rewriting the row data.
> This work is avoidable when the required bytes form contiguous runs of the
> parent's compression chunks.
> ## Proposed approach
> Introduce reusable zero-copy SSTable-splitting machinery that:
> - Selects contiguous runs of compression chunks for each child SSTable.
> - Retains the compressed bytes verbatim without deserializing or
> recompressing rows.
> - Uses byte-range reflinks to share filesystem extents when supported.
> - Falls back to copying the compressed bytes when reflinks are unavailable.
> - Rebuilds the index, filter, summary, statistics, and other components from
> index-only passes.
> - Represents compression-chunk prefixes that do not contain indexed
> partitions, allowing boundary chunks to be retained safely.
> - Preserves the validation, corruption detection, lifecycle, disk-space, and
> compatibility guarantees expected by existing SSTable operations.
> The same machinery can then be used by anticompaction and range streaming
> instead of their existing row-by-row rewrite paths.
> ## Delivery plan
> The work will be delivered incrementally so the on-disk representation and
> splitting primitives can be validated before they are introduced into
> critical online paths.
> ### Phase 1: BIG-format splitting
> Add zero-copy splitting for compressed BIG-format SSTables and expose it
> through an opt-in `sstablesplit --zero-copy` mode.
> Using the offline splitter as the first caller provides a bounded way to
> validate chunk selection, component rebuilding, extent sharing,
> compatibility, verification, scrub, streaming, loading, and upgrade behavior.
> This phase is limited to BIG SSTables without SSTable-attached indexes.
> ### Phase 2: BTI-format splitting
> Add the equivalent zero-copy splitting implementation for BTI SSTables,
> including the format-specific index rebuilding required by BTI.
> ### Phase 3: Secondary-index support
> Add support for splitting SSTables that have Storage-Attached Indexes (SAI)
> or legacy secondary indexes (2i). This phase must ensure that index
> components are rebuilt, shared, or invalidated correctly for every generated
> child.
> ### Phase 4: Zero-copy anticompaction
> Integrate the splitting machinery with anticompaction so repaired, transient,
> and unrepaired children can retain existing compressed chunks instead of
> rewriting every row.
> The optimized anticompaction path will initially be disabled by default and
> retain the existing rewrite path as a fallback.
> ### Phase 5: Zero-copy range streaming
> Integrate the machinery with range streaming so eligible streams transfer
> compressed chunk runs together with enough derived information for the
> receiver to construct the resulting SSTables without deserializing and
> rewriting rows.
> The optimized streaming path will initially be disabled by default and retain
> the existing row-based path for compatibility and fallback.
> ## Compatibility and fallback
> The new representation and protocol behavior must be explicitly
> version-gated. Nodes must not create or transmit split-prefix SSTables to
> peers that cannot interpret them.
> Zero-copy operation must remain an optimization rather than a correctness
> requirement. Unsupported formats, components, indexes, filesystems, peer
> versions, or split layouts must either be rejected before modification or use
> an existing safe fallback path.
> Reflink support is optional. Filesystems without extent sharing still benefit
> from avoiding row deserialization and recompression by copying the existing
> compressed bytes.
> ## Expected outcome
> For eligible SSTables, anticompaction and range streaming will avoid
> row-level deserialization, serialization, and recompression. The remaining
> work will primarily be compression-chunk transfer or extent sharing plus
> rebuilding derived components from indexes.
> This should substantially reduce CPU use, write amplification, temporary disk
> requirements, and elapsed time for both operations while preserving the
> existing behavior as a fallback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
