[
https://issues.apache.org/jira/browse/HDDS-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-17100:
---------------------------------
Component/s: Ozone Client
Assignee: Siyao Meng
> Do not send chunk checksums in ReadChunk requests
> -------------------------------------------------
>
> Key: HDDS-17100
> URL: https://issues.apache.org/jira/browse/HDDS-17100
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> {{ChunkInputStream#getChunkInfo}} builds the {{ChunkInfo}} of every ReadChunk
> request by copying the chunk's full {{ChunkInfo}}, including the checksum of
> every {{bytesPerChecksum}} slice of the chunk (256 checksums for a 4 MB chunk
> at 16 KB per checksum) and, for EC, the stripe checksum. This happens even
> for a small positioned read.
> The datanode does not use these checksums for client reads (it only verifies
> data read for Ratis, see HDDS-5623), but it parses them and echoes the
> request {{ChunkInfo}} back in the response, which the client parses again.
> The client verifies the data against its own copy of the chunk checksums, not
> the echoed ones.
> On a 3 datanode docker compose cluster with random 64 KB preads from 8
> threads, parsing the ReadChunk request was about 12% of datanode CPU, almost
> all of it in {{ChecksumData$Builder.mergeFrom}}. Leaving the checksums out of
> the request cut datanode CPU per read from about 58 µs to 34 µs (request
> parsing dropped to 2.6% of datanode CPU). Latency on the idle test cluster
> was unchanged within noise.
> Proposed fix: clear {{checksums}} (keeping type and {{bytesPerChecksum}}) and
> {{stripeChecksum}} in the ReadChunk request {{ChunkInfo}}. The change is
> client side only and works with existing datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
