[
https://issues.apache.org/jira/browse/HDDS-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HDDS-16377:
-----------------------------------
Parent: HDDS-10338
Issue Type: Sub-task (was: Bug)
> Datanode ReadBlock streaming path allocates and copies a fresh buffer for
> every request
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-16377
> URL: https://issues.apache.org/jira/browse/HDDS-16377
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Affects Versions: 2.3.0
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
>
> When a client reads a key through the streaming ReadBlock path, the datanode
> serves the block in pieces of about 1 MiB, one gRPC request per piece. For
> every such request KeyValueHandler.readBlockImpl does three wasteful things.
> First, it allocates a brand new 1 MiB heap ByteBuffer as a scratch buffer,
> only to overwrite it right away with data read from the block file. The JVM
> has to zero this memory on every allocation. For an 80 GiB key that is about
> 82000 allocations and 82000 memset passes over 1 MiB each.
> Second, ContainerCommandResponseBuilders.getReadBlockResponse turns that
> buffer into the response with ByteString.copyFrom. This allocates and zeros
> another 1 MiB byte array and then copies the data into it. The copy is not
> needed. gRPC serializes the response on the calling thread inside onNext, so
> by the time readBlockImpl comes back and refills the scratch buffer, the
> bytes are already in the network write buffers. Wrapping the buffer instead
> of copying it is safe.
> Third, each request looks up and decodes the BlockData for the block again
> from RocksDB, even though a read stream always serves the same single block
> and the metadata rarely changes between requests.
> An async-profiler run on the read path showed that the first two items
> together account for more than half of the CPU samples on the datanode
> request threads: about 34 percent in the copy inside getReadBlockResponse and
> about 21 percent in the per-request buffer allocation. This is pure overhead
> of memory allocation, zeroing and copying that does not move any data to the
> client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
