[
https://issues.apache.org/jira/browse/HDDS-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-16783:
---------------------------------
Assignee: Siyao Meng
> Support FileSystem#setVerifyChecksum in ofs and o3fs
> ----------------------------------------------------
>
> Key: HDDS-16783
> URL: https://issues.apache.org/jira/browse/HDDS-16783
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Filesystem
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h3. Problem
> HBase stores its own checksums in HFile blocks. When
> {{hbase.regionserver.checksum.verify}} is true (the default), {{HFileSystem}}
> opens a second FileSystem instance, calls {{setVerifyChecksum(false)}} on it,
> and reads HFile blocks through that instance. On HDFS this skips the file
> system checksum.
> {{BasicOzoneFileSystem}} and {{BasicRootedOzoneFileSystem}} do not override
> {{setVerifyChecksum}}, so the call reaches the empty default in
> {{FileSystem}} and the Ozone client still verifies every read. This costs
> HBase twice:
> * the data is checksummed by both Ozone and HBase
> * with verification on, each read is widened to checksum boundaries
> ({{ozone.client.bytes.per.checksum}}), so with a 1MB value a 64KB HFile block
> read fetches and checksums 1MB
> {{ozone.client.verify.checksum=false}} is not a good substitute. It applies
> to the whole process, including the FileSystem instance HBase falls back to
> when an HFile checksum fails, so that retry would no longer be verified by
> anything.
> h3. Proposal
> Override {{setVerifyChecksum}} in both file systems and pass the flag through
> {{OzoneClientAdapter}} and {{ClientProtocol}} to the client's own
> {{OzoneClientConfig}}, so that input streams opened afterwards by that
> FileSystem instance skip checksum verification. The setting is per instance,
> the same as {{DistributedFileSystem}}.
> No protocol change is needed. The existing read paths already read this flag
> when a stream is opened: ReadChunk, short-circuit read, and streaming
> ReadBlock, which passes it to the datanode as {{includeChecksums}}
> (HDDS-16258).
> h3. Expected gain
> 64KB random positional reads ({{readFully(pos, buf, 0, len)}} on one shared
> input stream per file), 16 threads, operations per second, median of three
> rounds. "Verify off" was measured with
> {{ozone.client.verify.checksum=false}}, which is what the API call would
> toggle per instance.
> ||Read path||Verify on||Verify off||
> |gRPC, 1MB bytes per checksum|6,100|22,600|
> |gRPC, 16KB bytes per checksum|22,100|22,600|
> |Short-circuit read, 1MB bytes per checksum|38,500|937,800|
> |Short-circuit read, 16KB bytes per checksum|534,900|911,400|
> Environment:
> * Ozone 2.3.0-SNAPSHOT, master at 24cf9aaa28e, Hadoop 3.4.3, Ratis 3.3.1
> * MiniOzoneCluster with 3 datanodes in one JVM ({{-Xmx8192m}}, G1), 16 files
> of 64MB, RATIS/THREE
> * {{apache/ozone-runner:20260626-1-jdk25}} container (Rocky Linux 9.8,
> OpenJDK 25.0.2), with native libhadoop for Hadoop 3.4.3 (linux aarch64) for
> short-circuit read
> * Docker Desktop Linux VM on arm64 with 14 CPUs and 23.4 GB, on an Apple M4
> Pro host (14 cores, 48 GB)
> * 3 s warm up and 8 s per measurement, host at least 89% idle at the start of
> each run
> * Single machine, loopback network, all data in the page cache, security off.
> Treat the figures as an upper bound.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
