[
https://issues.apache.org/jira/browse/HDDS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855468#comment-17855468
]
Chung En Lee commented on HDDS-8942:
------------------------------------
The issue occurs when the follower state machine hasn't finished applying log .
The BlockInputStream reads data from one of the datanodes in the pipeline. If
the BlockInputStream read data from follower before the statemachine finished
updating, the BlockInputStream read may be incorrect.
I tried to make the BlockInputStream read data from leader first. Here is the
changes:
{code:java}
if (request.getCmdType() == ContainerProtos.Type.GetBlock) {
blockID = request.getGetBlock().getBlockID();
+ datanodeList = pipeline.getNodes();
+ int getBlockDNLeaderIndex =
datanodeList.indexOf(pipeline.getLeaderNode());
+ if (getBlockDNLeaderIndex > 0) {
+ // Pull the Cached DN to the top of the DN list
+ Collections.swap(datanodeList, 0, getBlockDNLeaderIndex);
+ }
} else if (request.getCmdType() == ContainerProtos.Type.ReadChunk) {
blockID = request.getReadChunk().getBlockID();
} else if (request.getCmdType() == ContainerProtos.Type.GetSmallFile) {
blockID = request.getGetSmallFile().getBlock().getBlockID();
}
- if (blockID != null) {
+ if (blockID != null && datanodeList != null) {
// Check if the DN to which the GetBlock command was sent has been
cached.
DatanodeDetails cachedDN = getBlockDNcache.get(blockID);
if (cachedDN != null) {
{code}
And run {{TestHSync}}
[https://github.com/chungen0126/ozone/actions/runs/9532695490]
> Intermittent failure in ITestOzoneContractCreate#testSyncable
> -------------------------------------------------------------
>
> Key: HDDS-8942
> URL: https://issues.apache.org/jira/browse/HDDS-8942
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Attila Doroszlai
> Assignee: Raju Balpande
> Priority: Major
> Attachments: image-2024-05-17-20-20-02-892.png,
> org.apache.hadoop.fs.ozone.contract.AbstractOzoneContractTest$TestContractCreate-output.txt,
>
> org.apache.hadoop.fs.ozone.contract.AbstractOzoneContractTest$TestContractCreate.txt
>
>
> {code}
> ITestOzoneContractCreate.testSyncable Time elapsed: 0.277 s <<< ERROR!
> java.io.IOException: Inconsistent read for blockID=conID: 7 locID:
> 111677748019200007 bcsId: 0 length=2 position=1 numBytesToRead=1
> numBytesRead=-1
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.checkPartBytesRead(KeyInputStream.java:183)
> at
> org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:97)
> at
> org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
> at
> org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:43)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:55)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.validateSyncableSemantics(AbstractContractCreateTest.java:548)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testSyncable(AbstractContractCreateTest.java:459)
> {code}
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/02/03/19930/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractCreate.txt
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/30/22797/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractCreate.txt
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/20/23549/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractCreate.txt
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/27/23809/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractCreate.txt
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/08/31/25075/it-contract/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractCreate.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
