Steve Loughran created HADOOP-18391:
---------------------------------------
Summary: harden VectoredReadUtils
Key: HADOOP-18391
URL: https://issues.apache.org/jira/browse/HADOOP-18391
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 3.3.9
Reporter: Steve Loughran
Assignee: Mukund Thakur
harden the VectoredReadUtils methods for consistent and more robust use,
especially in those filesystems which don't have the api.
VectoredReadUtils.readInDirectBuffer should allocate a max buffer size, .e.g
4mb, then do repeated reads and copies; this ensures that you don't OOM with
many threads doing ranged requests. other libs do this.
readVectored to call validateNonOverlappingAndReturnSortedRanges before
iterating
this ensures the abfs/s3a requirements are always met, and that because ranges
will be read in order, prefetching by other clients will keep their performance
good.
readVectored to add special handling for 0 byte ranges
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
