Kirill Sizov created HDDS-11223:
-----------------------------------
Summary: Datanode checksum validation in EC bucket does not work
Key: HDDS-11223
URL: https://issues.apache.org/jira/browse/HDDS-11223
Project: Apache Ozone
Issue Type: Bug
Reporter: Kirill Sizov
*Steps to reproduce*
- Enable hdds.datanode.chunk.data.validation.check property.
- put any file into a EC bucket
The output a client gets is:
{noformat}
INTERNAL_ERROR No enough datanodes to choose. TotalNodes = 10 AvailableNodes =
4 RequiredNodes = 5 ExcludedNodes = 6 UsedNodes = 0
{noformat}
*Investigation details*
The issue comes from incorrect checksum validation on the datanode side.
Here are the details:
1. {{KeyValueHandler.validateChunkChecksumData}} calls
{{Checksum.verifyChecksum}}
2. {{Checksum.verifyChecksum}} calls {{checksum.computeChecksum(data)}} to
calculate a checksum over the received data
3. {{checksum.computeChecksum}} has the following block:
{code}
for (ByteBuffer b : data.iterate(bytesPerChecksum)) {
checksumList.add(computeChecksum(b, function, bytesPerChecksum));
}
{code}
The issue is that {{data.iterate()}} may return byte buffers of an arbitrary
size.
Specifically, it is observed on {{ChunkBufferImplWithByteBufferList}} which was
created from a {{RopeByteString}} in {{KeyValueHandler.handleWriteChunk}}:
{code}
data =
ChunkBuffer.wrap(writeChunk.getData().asReadOnlyByteBufferList());
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
