Chu Cheng Li created HDDS-17317:
-----------------------------------
Summary: Send the block data of Ratis streaming reads with sendfile
Key: HDDS-17317
URL: https://issues.apache.org/jira/browse/HDDS-17317
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode
Reporter: Chu Cheng Li
Assignee: Chu Cheng Li
*Background*
In a Ratis streaming read (HDDS-9904), the datanode sends each ReadBlock reply
as one data stream packet. The packet contains the length of the response, the
response with the chunk info and the checksums, and then the data.
*Problem*
The datanode reads the data from the block file into a buffer. Then netty
writes the buffer to the socket. Thus, the datanode copies each byte two times:
from the page cache into the buffer, and from the buffer into the socket. HDFS
sends block data with {{transferTo}} (sendfile), without these copies.
*Effect*
On one host with 3 datanodes on one NVMe disk, one TestDFSIO read stream reads
0.8 to 0.95 GB/s. HDFS reads 2.3 to 2.5 GB/s on the same host.
* In a cold 8 GB read, the datanode request threads use 4.9 CPU seconds. The
block file read uses 3.0 seconds of this.
* The client waits for the next reply for 78% of its read time.
*Changes*
* Add the datanode setting
{{hdds.container.ratis.datastream.read.sendfile.enabled}}. The default is false.
* When the setting is true, {{KeyValueHandler}} gives the block file, the
offset and the length of each reply to the observer. It does not read the data.
* {{ContainerStateMachine}} sends the response as the header of the packet, and
the data from the block file with sendfile (RATIS-2773). It waits until all the
packets are sent before it closes the block file.
* The packet format does not change. The response still contains the checksums.
Thus, the client does not change, and it still verifies the checksums.
* The datanode does not use sendfile when
{{hdds.datanode.chunk.data.validation.check}} is true, or when the data stream
uses TLS.
*Results*
The prototype was tested with TestDFSIO on one host, with 3 datanodes on one
NVMe disk. The values are MB/s for each file, the median of 3 runs. The read
window is the default 8 MB.
||Test||Buffer copy||sendfile||
|1 file of 8 GB, cold|812|1,152|
|1 file of 8 GB, warm|949|1,251|
|16 files of 4 GB, cold|273|285|
|16 files of 4 GB, warm|502|555|
|Datanode CPU seconds per GB, 16 files, cold|2.79|1.50|
|Datanode CPU seconds per GB, 16 files, warm|2.04|0.78|
* The datanodes restart before each mode. Thus, the CPU values include the JVM
start in both modes.
* With sendfile, a larger read window also helps one stream. With
{{ozone.client.ratis.stream.read.window-size}} at 32 MB, one stream reads about
1.6 GB/s with sendfile. It reads 1.05 to 1.13 GB/s with the buffer copy.
* HDFS uses 0.42 to 0.74 datanode CPU seconds per GB in the same tests.
* With 16 files, the disk (cold) and the client (warm) limit the throughput.
Thus, the gain is in the datanode CPU.
*Open points*
* sendfile runs in the netty event loop. If a page is not in the page cache,
the event loop waits for the disk. A readahead with
{{posix_fadvise(WILLNEED)}}, as HDFS does, can prevent this.
* With sendfile, the client limits one stream. The netty receive path of the
client uses about 0.9 CPU seconds per GB. The default 8 MB window keeps only 2
requests in flight.
This change needs RATIS-2773 in a Ratis release.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
