Andrey Yarovoy created HDDS-17865:
-------------------------------------

             Summary: Datanode GetBlock decodes and re-encodes the whole stored 
BlockData, including every chunk's checksum list
                 Key: HDDS-17865
                 URL: https://issues.apache.org/jira/browse/HDDS-17865
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Datanode
            Reporter: Andrey Yarovoy


h3. Problem

For every GetBlock, the datanode parses the stored block metadata from RocksDB 
and then serializes the same data back into the response:

{noformat}
KeyValueHandler.handleGetBlock
BlockManagerImpl.getBlock -> DatanodeStore.getBlockByID -> 
blockDataTable.get(blockKey)
BlockData codec (DelegatedCodec): ContainerProtos.BlockData.parseFrom(bytes) -> 
BlockData.getFromProtoBuf
BlockData.getProtoBufMessage() computeSize(chunks) check, new builder, 
addAllChunks
responseData.getSerializedSize() for the bytesGetBlock metric
gRPC / domain socket serializes the response again
{noformat}

{\{getBlock}} itself needs only the block commit sequence ID from the decoded 
object, for the BCSID check. The response bytes are effectively the bytes that 
were stored.

The decode is expensive because each \{{ChunkInfo}} carries a \{{ChecksumData}} 
with one \{{repeated bytes checksums}} entry per 
\{{ozone.client.bytes.per.checksum}} (default 16 KB, CRC32). A 4 MB chunk has 
256 checksums, and a full 256 MB block has about 16k \{{ByteString}} objects to 
allocate on every GetBlock.

h3. Evidence

async-profiler CPU profiles of datanodes serving HBase YCSB workload E (scan). 
HBase reopens store files when it switches a scanner to stream read, and every 
reopen issues a fresh GetBlock per block.

||Profile||GetBlock, % of datanode CPU||Inside GetBlock||
|Ratis RATIS/THREE + short-circuit read|28.7% (75% of 
\{{HddsDispatcher.dispatch}} CPU)|BlockData parse 58% 
(\{{ChecksumData$Builder.mergeFrom}}, of which \{{ProtobufArrayList.add}} -> 
\{{Arrays.copyOf}} 49%), response \{{getSerializedSize}} 11%, RocksDB 
\{{getDirect}} 14%, \{{getShortCircuitFd}} 5%|
|EC rs-3-2|11.2%|proto decode/re-encode 9.3% of datanode CPU, RocksDB get 2.6%|

On the RegionServer, GetBlock is 11% (EC) and 19.6% (Ratis) of busy handler 
time on the scan path. Part of that is the network round trip, but the 
datanode-side cost is mostly this decode/re-encode. The datanode itself is not 
saturated (dispatcher busy below 1% of thread time), so the gain is CPU and 
datanode headroom, plus a shorter GetBlock round trip.

h3. Proposed fix

Serve the stored bytes without a full decode. The wire format stays exactly the 
same.

# Read the raw value of the block key (\{{RDBTable.get(byte[])}}), skipping the 
BlockData codec.
# Partially parse the raw bytes with \{{CodedInputStream}}. Read \{{blockID}} 
(for the BCSID check) and \{{metadata}} (to detect 
\{{INCREMENTAL_CHUNK_LIST}}), and skip every \{{chunks}} field without decoding 
it.
# If the block uses an incremental chunk list (hsync, last chunk kept in 
\{{lastChunkInfoTable}}), fall back to the current path, which merges the last 
chunk.
# Otherwise put the raw bytes into \{{GetBlockResponseProto}} field 1 
(\{{blockData}}) as an already-encoded length-delimited field, for example 
through the builder's unknown-field set followed by \{{buildPartial()}}. The 
serialized response is then byte-for-byte what it is today, and clients parse 
it unchanged.
# Use the raw length for the \{{bytesGetBlock}} and \{{localBytesGetBlock}} 
metrics instead of \{{getSerializedSize()}}.

Alternatives considered:
* *Cache decoded BlockData on the datanode.* A full block's decoded checksum 
list costs hundreds of KB of heap, so a useful cache is large. Caching the raw 
bytes instead is cheap, but it pays off only together with step 4.
* *\{{[lazy = true]}} on \{{ChunkInfo.checksumData}}.* This is a proto 
annotation only, so it is wire compatible. Whether the shaded protobuf-java 
runtime honors it for proto2 singular message fields needs checking before 
relying on it.

h3. Interactions

* HDDS-16333 (open) asks GetBlock to validate \{{ChunkInfo}}. Validation on the 
read path needs the decode this change removes. Validating once at PutBlock 
time, or only on the fallback path, keeps both.
* HDDS-17112 removed a similar decode from PutBlock with an incremental chunk 
list. This is the read-side counterpart.
* The client still parses the full response once per GetBlock. That cost is out 
of scope here.

h3. Compatibility

No proto change, no RocksDB layout change, wire-identical responses. Blocks 
with an incremental chunk list keep the current code path.

h3. Testing

* Unit test: for blocks with 1, 64 and 128 chunks (256 checksums each), the new 
response bytes equal the bytes of the current \{{getProtoBufMessage()}} 
response. Also cover a BCSID mismatch, a missing block, and the 
incremental-chunk-list fallback.
* Microbenchmark of \{{handleGetBlock}} with a 64-chunk block, before and 
after. The profile predicts about 70% less GetBlock CPU.
* Re-profile the datanode under YCSB E. On the Ratis cluster, GetBlock should 
drop from about 29% to below 10% of datanode CPU.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to