[
https://issues.apache.org/jira/browse/HADOOP-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642456#comment-17642456
]
ASF GitHub Bot commented on HADOOP-18543:
-----------------------------------------
steveloughran commented on PR #5172:
URL: https://github.com/apache/hadoop/pull/5172#issuecomment-1335114105
> I'd like to make oss also implement openFile() in this pr as s3a does
which could also meet our needs.
This is exactly what the API was designed for -to let people provide extra
hints/options. to the object stores.
Do read the filesystem.md and related docs on the topic, and look at the
abfs implementation as well as the s3a one.
If your implementation takes a FileStatus or length option, it can then skip
the HEAD request on opening and save time and money. All the hadoop internal
uses of openFile() do this. My own copies of the parquet and avro readers also
do it for better cloud reading performance
> AliyunOSS: AliyunOSSFileSystem#open(Path path, int bufferSize) should use
> buffer size as its downloadPartSize
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18543
> URL: https://issues.apache.org/jira/browse/HADOOP-18543
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/oss
> Reporter: Hangxiang Yu
> Priority: Major
> Labels: pull-request-available
>
> In our application, different components have their own suitable buffer size
> to download.
> But currently, AliyunOSSFileSystem#open(Path path, int bufferSize) just get
> downloadPartSize from configuration.
> We cannnot use different value for different components in our programs.
> I think we should the method should use the buffer size from the paramater.
> AliyunOSSFileSystem#open(Path path) could have default value as current
> default downloadPartSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
