[
https://issues.apache.org/jira/browse/HDDS-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17865:
-------------------------------------
Assignee: Andrey Yarovoy
> Datanode GetBlock decodes and re-encodes the whole stored BlockData,
> including every chunk's checksum list
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17865
> URL: https://issues.apache.org/jira/browse/HDDS-17865
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h3. Problem
> For every GetBlock, the datanode parses the stored block metadata from
> RocksDB and then serializes the same data back into the response:
> {noformat}
> KeyValueHandler.handleGetBlock
> BlockManagerImpl.getBlock -> DatanodeStore.getBlockByID ->
> blockDataTable.get(blockKey)
> BlockData codec (DelegatedCodec): ContainerProtos.BlockData.parseFrom(bytes)
> -> BlockData.getFromProtoBuf
> BlockData.getProtoBufMessage() computeSize(chunks) check, new builder,
> addAllChunks
> responseData.getSerializedSize() for the bytesGetBlock metric
> gRPC / domain socket serializes the response again
> {noformat}
> {\{getBlock}} itself needs only the block commit sequence ID from the decoded
> object, for the BCSID check. The response bytes are effectively the bytes
> that were stored.
> The decode is expensive because each \{{ChunkInfo}} carries a
> \{{ChecksumData}} with one \{{repeated bytes checksums}} entry per
> \{{ozone.client.bytes.per.checksum}} (default 16 KB, CRC32). A 4 MB chunk has
> 256 checksums, and a full 256 MB block has about 16k \{{ByteString}} objects
> to allocate on every GetBlock.
> h3. Evidence
> async-profiler CPU profiles of datanodes serving HBase YCSB workload E
> (scan). HBase reopens store files when it switches a scanner to stream read,
> and every reopen issues a fresh GetBlock per block.
> ||Profile||GetBlock, % of datanode CPU||Inside GetBlock||
> |Ratis RATIS/THREE + short-circuit read|28.7% (75% of
> \{{HddsDispatcher.dispatch}} CPU)|BlockData parse 58%
> (\{{ChecksumData$Builder.mergeFrom}}, of which \{{ProtobufArrayList.add}} ->
> \{{Arrays.copyOf}} 49%), response \{{getSerializedSize}} 11%, RocksDB
> \{{getDirect}} 14%, \{{getShortCircuitFd}} 5%|
> |EC rs-3-2|11.2%|proto decode/re-encode 9.3% of datanode CPU, RocksDB get
> 2.6%|
> On the RegionServer, GetBlock is 11% (EC) and 19.6% (Ratis) of busy handler
> time on the scan path. Part of that is the network round trip, but the
> datanode-side cost is mostly this decode/re-encode. The datanode itself is
> not saturated (dispatcher busy below 1% of thread time), so the gain is CPU
> and datanode headroom, plus a shorter GetBlock round trip.
> h3. Proposed fix
> Serve the stored bytes without a full decode. The wire format stays exactly
> the same.
> # Read the raw value of the block key (\{{RDBTable.get(byte[])}}), skipping
> the BlockData codec.
> # Partially parse the raw bytes with \{{CodedInputStream}}. Read \{{blockID}}
> (for the BCSID check) and \{{metadata}} (to detect
> \{{INCREMENTAL_CHUNK_LIST}}), and skip every \{{chunks}} field without
> decoding it.
> # If the block uses an incremental chunk list (hsync, last chunk kept in
> \{{lastChunkInfoTable}}), fall back to the current path, which merges the
> last chunk.
> # Otherwise put the raw bytes into \{{GetBlockResponseProto}} field 1
> (\{{blockData}}) as an already-encoded length-delimited field, for example
> through the builder's unknown-field set followed by \{{buildPartial()}}. The
> serialized response is then byte-for-byte what it is today, and clients parse
> it unchanged.
> # Use the raw length for the \{{bytesGetBlock}} and \{{localBytesGetBlock}}
> metrics instead of \{{getSerializedSize()}}.
> Alternatives considered:
> * *Cache decoded BlockData on the datanode.* A full block's decoded checksum
> list costs hundreds of KB of heap, so a useful cache is large. Caching the
> raw bytes instead is cheap, but it pays off only together with step 4.
> * *\{{[lazy = true]}} on \{{ChunkInfo.checksumData}}.* This is a proto
> annotation only, so it is wire compatible. Whether the shaded protobuf-java
> runtime honors it for proto2 singular message fields needs checking before
> relying on it.
> h3. Interactions
> * HDDS-16333 (open) asks GetBlock to validate \{{ChunkInfo}}. Validation on
> the read path needs the decode this change removes. Validating once at
> PutBlock time, or only on the fallback path, keeps both.
> * HDDS-17112 removed a similar decode from PutBlock with an incremental chunk
> list. This is the read-side counterpart.
> * The client still parses the full response once per GetBlock. That cost is
> out of scope here.
> h3. Compatibility
> No proto change, no RocksDB layout change, wire-identical responses. Blocks
> with an incremental chunk list keep the current code path.
> h3. Testing
> * Unit test: for blocks with 1, 64 and 128 chunks (256 checksums each), the
> new response bytes equal the bytes of the current \{{getProtoBufMessage()}}
> response. Also cover a BCSID mismatch, a missing block, and the
> incremental-chunk-list fallback.
> * Microbenchmark of \{{handleGetBlock}} with a 64-chunk block, before and
> after. The profile predicts about 70% less GetBlock CPU.
> * Re-profile the datanode under YCSB E. On the Ratis cluster, GetBlock should
> drop from about 29% to below 10% of datanode CPU.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]