[
https://issues.apache.org/jira/browse/HDDS-16805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16805:
------------------------------
Attachment: TestBugMC7ReconcileUnverifiedPeerChunk.java
> Container reconciliation does not verify chunks pulled from a peer when
> ozone.client.verify.checksum is false on the datanode
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16805
> URL: https://issues.apache.org/jira/browse/HDDS-16805
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-7-verify-peer-chunk-checksums-in-reconciliation.patch,
> TestBugMC7ReconcileUnverifiedPeerChunk.java
>
>
> h3. Mechanism
> {{KeyValueHandler.reconcileChunksPerBlock}} reads chunks from a peer through
> a {{BlockInputStream}} that it creates with
> {{conf.getObject(OzoneClientConfig.class)}}, the datanode's own view of the
> client configuration. The client streams verify what they read only when
> {{ozone.client.verify.checksum}} is true, and the peer does not verify on
> {{ReadChunk}}. With the setting false in the datanode's configuration, a
> chunk that is corrupt on the peer is written into the local replica, the
> local merkle tree records it as healthy with the checksum from the peer's
> block metadata, and the block is committed with the peer's BCSID. A chunk
> that the local tree lists as healthy is never pulled, so this affects blocks
> and chunks that the replica lacks or that its own scanner marked corrupt.
> The peer's tree is only as fresh as the peer's last scan, so a chunk that
> went bad since then is still listed as healthy and offered for repair.
> The reconciliation design document says that the datanode validates the
> checksum of a chunk read from a peer before using it, and that the existing
> client code already does this. That holds only while the client setting is
> on. {{ECReconstructionCoordinator}}, the other datanode path that reads peer
> data through the client streams, calls {{setChecksumVerify(true)}} on its
> config before it opens the stream. The line in {{reconcileChunksPerBlock}}
> dates from HDDS-11763 (read from history).
> h3. Trigger
> # {{ozone.client.verify.checksum=false}} is in the configuration that the
> datanode loads, for example a site file shared with clients.
> # A replica lacks a block or chunk that a peer has, or holds a chunk that its
> own scanner marked corrupt.
> # On the peer that chunk is corrupt on disk and the peer has not been scanned
> since.
> # An administrator runs {{ozone admin container reconcile}} for the container.
> h3. Impact
> * The corrupt chunk is copied into the reconciling replica and committed. A
> later round against a healthy peer in the same reconcile task does not repair
> it, because the local tree already lists the chunk as healthy: after
> reconciling with the corrupt peer and then a healthy one, the block still
> holds the corrupt bytes (run).
> * The on demand scan at the end of the reconcile task finds the mismatch and
> marks the replica UNHEALTHY. In the reproduction a replica that was CLOSED
> and only lacked one block before the reconcile is UNHEALTHY after it (run).
> * With the default setting the corrupt chunk is rejected with a checksum
> mismatch and the block is repaired from the healthy peer (run, as the
> control).
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC7ReconcileUnverifiedPeerChunk.java]
> ({{hadoop-hdds/container-service}}) uses the setup of
> {{TestContainerReconciliationWithMockDatanodes}}: three datanodes, each a
> real {{KeyValueHandler}} with a real container on disk and a real on demand
> scanner, the real {{BlockInputStream}}, and only the static
> {{ContainerProtocolCalls}} transport routed to the peer's handler. There is
> no SCM. One replica loses a block, one peer has a byte flipped in that block
> and is not scanned, the other peer is healthy. The replica reconciles with
> the corrupt peer first and the healthy peer second. With the setting false on
> the reconciling datanode the block is committed with the peer's BCSID, the
> block file equals the corrupt peer's bytes, and the replica is UNHEALTHY
> after the scan. With the setting true the block file equals the healthy bytes
> and the replica stays CLOSED. A passing test means the defect is present.
> h3. Patch
> [^MC-7-verify-peer-chunk-checksums-in-reconciliation.patch], against
> 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at
> 8ca44914771.
> {{reconcileChunksPerBlock}} calls {{setChecksumVerify(true)}} on the
> {{OzoneClientConfig}} that it passes to the stream, as
> {{ECReconstructionCoordinator}} does. {{getObject}} returns a new object on
> every call, so no other code sees the flag. Nothing changes with the default
> setting. With the setting false, reconciliation now computes checksums over
> what it pulls, and a chunk that fails ends the repair of that block from that
> peer, which is the existing handling of a chunk that cannot be pulled. The
> next peer is then tried.
> Covered by
> {{testContainerReconciliationVerifiesPeerDataWhenClientChecksumVerifyDisabled}}
> in the existing {{TestContainerReconciliationWithMockDatanodes}}. Without
> the change it fails because the data checksum of the replica after reconcile
> and scan is not the healthy one. With it the unit suites of the {{keyvalue}},
> {{ozoneimpl}} and {{checksum}} packages in {{hadoop-hdds/container-service}}
> pass (1124 tests, 18 skipped by the suites), checkstyle is clean, and the
> case of the reproduction with the setting false fails because the block now
> holds the healthy bytes. Not run on a mini cluster.
> Found by TLA+ model checking and code review of the container reconciliation
> paths in the datanode and SCM under HDDS-15926, on commit
> 1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
