[
https://issues.apache.org/jira/browse/HDDS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chung-En Lee updated HDDS-15886:
--------------------------------
Description:
With `ozone.client.stream.read.block` enabled, a reader that opens a file that
is still being written can't read all the data the writer has hsync'ed. It hits
EOF at the block length OM last recorded. With the setting disabled, the same
read succeeds.
Why OM's block length is out of date
To keep OM load down, BlockOutputStreamEntryPool#hsyncKey only sends the new
length to OM when the last block has changed. Later hsyncs that stay within the
same block write the data to the datanodes but don't update OM. So for the last
block of an hsync'ed file, OM's length can be shorter than what is actually on
the datanode.
The read path expects this. KeyInputStream marks the last block of an hsync'ed
file as under construction (setUnderConstruction(true)). BlockInputStream then
gets the real length from the datanode.
StreamBlockInputStream doesn't do this. It takes blockInfo.getLength() from OM
through BlockInputStreamFactoryImpl, stores it in a final field, and never
checks isUnderConstruction().
This issue was found by `TestContractCreate#testSyncable` . It fails with
streamReadBlock enabled.
was:
*Issue:* During active writes using {{{}hsync{}}}, the actual block length on
the storage node keeps increasing. However, the client-side read stream only
uses a static block length cached at the beginning.
Because this cached length is never updated, the reader prematurely assumes it
has reached the end of the file (EOF) and fails to read any newly synced data.
See `MultiPartInputStream#readWithStrategy`.
*Expected Behavior:* The read stream should dynamically refresh or query the
latest block length when {{hsync}} is active, ensuring all synced data is fully
readable.
> Client unable to read newly synced data due to outdated block length
> --------------------------------------------------------------------
>
> Key: HDDS-15886
> URL: https://issues.apache.org/jira/browse/HDDS-15886
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Chung-En Lee
> Assignee: Chung-En Lee
> Priority: Critical
>
> With `ozone.client.stream.read.block` enabled, a reader that opens a file
> that is still being written can't read all the data the writer has hsync'ed.
> It hits EOF at the block length OM last recorded. With the setting disabled,
> the same read succeeds.
> Why OM's block length is out of date
> To keep OM load down, BlockOutputStreamEntryPool#hsyncKey only sends the new
> length to OM when the last block has changed. Later hsyncs that stay within
> the same block write the data to the datanodes but don't update OM. So for
> the last block of an hsync'ed file, OM's length can be shorter than what is
> actually on the datanode.
> The read path expects this. KeyInputStream marks the last block of an
> hsync'ed file as under construction (setUnderConstruction(true)).
> BlockInputStream then gets the real length from the datanode.
> StreamBlockInputStream doesn't do this. It takes blockInfo.getLength() from
> OM through BlockInputStreamFactoryImpl, stores it in a final field, and never
> checks isUnderConstruction().
> This issue was found by `TestContractCreate#testSyncable` . It fails with
> streamReadBlock enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
