Sergey Soldatov created HDDS-16377:
--------------------------------------
Summary: Datanode ReadBlock streaming path allocates and copies a
fresh buffer for every request
Key: HDDS-16377
URL: https://issues.apache.org/jira/browse/HDDS-16377
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 2.3.0
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
When a client reads a key through the streaming ReadBlock path, the datanode
serves the block in pieces of about 1 MiB, one gRPC request per piece. For
every such request KeyValueHandler.readBlockImpl does three wasteful things.
First, it allocates a brand new 1 MiB heap ByteBuffer as a scratch buffer,
only to overwrite it right away with data read from the block file. The JVM has
to zero this memory on every allocation. For an 80 GiB key that is about 82000
allocations and 82000 memset passes over 1 MiB each.
Second, ContainerCommandResponseBuilders.getReadBlockResponse turns that
buffer into the response with ByteString.copyFrom. This allocates and zeros
another 1 MiB byte array and then copies the data into it. The copy is not
needed. gRPC serializes the response on the calling thread inside onNext, so by
the time readBlockImpl comes back and refills the scratch buffer, the bytes are
already in the network write buffers. Wrapping the buffer instead of copying it
is safe.
Third, each request looks up and decodes the BlockData for the block again
from RocksDB, even though a read stream always serves the same single block and
the metadata rarely changes between requests.
An async-profiler run on the read path showed that the first two items
together account for more than half of the CPU samples on the datanode request
threads: about 34 percent in the copy inside getReadBlockResponse and about 21
percent in the per-request buffer allocation. This is pure overhead of memory
allocation, zeroing and copying that does not move any data to the client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
