FrankChen021 opened a new issue, #20768:
URL: https://github.com/apache/druid/issues/20768
### Use case
We want to evaluate the V10 segment format and partial segment loading on a
few datasources in a shared production cluster: segment size, push/load time,
and query latency, compared with the same data in V9. Other datasources in the
cluster must not be affected.
Today this is not possible without a cluster-wide change:
- V10 is enabled only by the runtime property `druid.indexer.task.buildV10`,
which applies to every task.
- Deep storage packaging (`druid.storage.zip`,
`druid.storage.compressionFormat`; see #20767) is also global. Even if a task
builds V10, its segments are archived unless archiving is turned off for all
datasources. Archived V10 segments cannot be range-read.
### Problems with archiving V10 segments
The V10 format stores a segment as a single `druid.segment` file: a metadata
header followed by containers that hold the internal files. It is designed to
be read by byte range, which `SegmentRangeReader` uses for partial downloads.
Archiving it:
- **Prevents partial reads.** `rangeable` is set only on the unarchived push
path, and the load spec returns no range reader for a zip key, so loads fall
back to a full download and extraction.
- **Hits a 2 GB limit.** `CompressionUtils.zip` and the lz4 archiver reject
any single file larger than `Integer.MAX_VALUE`. V9 avoids this because its
smoosh chunks are capped at 2 GB, but a V10 `druid.segment` can be larger.
- **Compresses twice.** Column data inside V10 is already compressed per
`IndexSpec`, so the archive costs CPU on every push and load.
### Proposal
1. **Always push V10 segments unarchived.** Each pusher checks the segment's
binary version (`SegmentUtils.getVersionFromDir`). For V10, it ignores
`druid.storage.zip` / `druid.storage.compressionFormat`, uploads
`druid.segment` (and any external files) as-is, and marks the load spec
`rangeable`. V9 segments keep following the configuration.
- Upload to a temporary location and then rename, so a failed push leaves
no partial segment.
- Killers must delete unarchived V10 segments. For example, the HDFS
killer currently rejects any path that is not `.zip`/`.lz4`.
2. **Add a `buildV10` task context key.** It overrides
`druid.indexer.task.buildV10` in both directions for a single task, so V10 can
be enabled per supervisor or per ingestion spec. For SQL-based ingestion, it is
set in the query context and passed on to the worker tasks that build segments.
Together, these allow enabling V10 and partial reads on selected datasources
by changing only their supervisor or ingestion spec. The segment format then
determines its packaging, so there is no storage configuration to keep in sync,
and other datasources are unaffected.
### Compatibility
- **V9 segments:** unaffected.
- **Existing V10 segments:** those stored as archives keep loading, because
the puller infers the format from the load spec path.
- **Readers:** any service that loads V10 segments already has to run a
version that supports V10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
