[
https://issues.apache.org/jira/browse/HADOOP-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640599#comment-17640599
]
Steve Loughran commented on HADOOP-18543:
-----------------------------------------
often the buffer size passed in is small, say 16-32 kb. would that be too small
this is a great time to implement openFile() as s3a and abfs does, which lets
caller pass in a list of options, a file length/status (saves on the HEAD) and
a read policy (random, whole file, sequential).
all the open() calls in hadoop codebase now use this and pass in the read
policy, length if known, and we do this internally in our own avro jars for for
avro file reads so as to guarantee sequential reads of iceberg manifests even
in clusters with the s3a read policy == random
> AliyunOSS: AliyunOSSFileSystem#open(Path path, int bufferSize) should use
> buffer size as its downloadPartSize
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18543
> URL: https://issues.apache.org/jira/browse/HADOOP-18543
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/oss
> Reporter: Hangxiang Yu
> Priority: Major
> Labels: pull-request-available
>
> In our application, different components have their own suitable buffer size
> to download.
> But currently, AliyunOSSFileSystem#open(Path path, int bufferSize) just get
> downloadPartSize from configuration.
> We cannnot use different value for different components in our programs.
> I think we should the method should use the buffer size from the paramater.
> AliyunOSSFileSystem#open(Path path) could have default value as current
> default downloadPartSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
