chungen0126 commented on PR #10764: URL: https://github.com/apache/ozone/pull/10764#issuecomment-4976097347
Hi @szetszwo , @sodonnel ,During my review of the current Stream Read architecture, I identified a checksum calculation issue that affects both the client and server sides.This issue occurs when a client actively triggers a flush or hsync. In such cases, the resulting chunk size does not perfectly align with `bytesPerChunk`, even though the checksum for this chunk has already been generated. Consequently, we can no longer simply rely on the fixed bytesPerChunk length to determine which chunk corresponds to the checksum currently being calculated.Here is a breakdown of the issues on both sides: - Client-side Issue: Currently, the response returned by the DataNode does not contain any chunk-related information. As a result, the client cannot correctly partition the retrieved data to verify the checksums. - Server-side Issue: The server cannot return the correct set of checksums. For example, if bytesPerChecksum is 1024, and we have chunk1 (data: a) and chunk2 (data: b). Under the current logic, the response will only return the checksum data for chunk1. To properly verify checksums on the client side, I believe it is essential to return chunkInfo. If we go down this path, we should refine the structure of ReadBlockResponseProto, as the top-level checksumData might become redundant. Since modifying Protobuf definitions after a major release is highly problematic due to backward compatibility/wire-protocol locking, this issue could potentially block or impact the upcoming Ozone 2.2 release. I'd love to hear your thoughts on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
