[GitHub] [hadoop] masteryhx commented on a diff in pull request #5172: HADOOP-18543. AliyunOSSFileSystem#open(Path path, int bufferSize) use buffer size as its downloadPartSize
masteryhx commented on code in PR #5172:
URL: https://github.com/apache/hadoop/pull/5172#discussion_r1035962332
##
hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSInputStream.java:
##
@@ -57,18 +57,21 @@ public class AliyunOSSInputStream extends FSInputStream {
private ExecutorService readAheadExecutorService;
private Queue readBufferQueue = new ArrayDeque<>();
- public AliyunOSSInputStream(Configuration conf,
- ExecutorService readAheadExecutorService, int maxReadAheadPartNumber,
- AliyunOSSFileSystemStore store, String key, Long contentLength,
- Statistics statistics) throws IOException {
+ public AliyunOSSInputStream(
+ long downloadPartSize,
+ ExecutorService readAheadExecutorService,
+ int maxReadAheadPartNumber,
+ AliyunOSSFileSystemStore store,
+ String key,
+ Long contentLength,
+ Statistics statistics) throws IOException {
this.readAheadExecutorService =
-MoreExecutors.listeningDecorator(readAheadExecutorService);
+MoreExecutors.listeningDecorator(readAheadExecutorService);
this.store = store;
this.key = key;
this.statistics = statistics;
this.contentLength = contentLength;
-downloadPartSize = conf.getLong(MULTIPART_DOWNLOAD_SIZE_KEY,
-MULTIPART_DOWNLOAD_SIZE_DEFAULT);
+this.downloadPartSize = downloadPartSize;
Review Comment:
I think we could see different performance between uploading/requesting 4KB
and 4MB ?
In my some cases, some data are orgnazied with unit of ~16KB, and I will
read them randomly.
In this case, I am sure what I need is just these KB, more data will cost
more time and bandwidth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] masteryhx commented on a diff in pull request #5172: HADOOP-18543. AliyunOSSFileSystem#open(Path path, int bufferSize) use buffer size as its downloadPartSize
masteryhx commented on code in PR #5172:
URL: https://github.com/apache/hadoop/pull/5172#discussion_r1035582679
##
hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSInputStream.java:
##
@@ -57,18 +57,21 @@ public class AliyunOSSInputStream extends FSInputStream {
private ExecutorService readAheadExecutorService;
private Queue readBufferQueue = new ArrayDeque<>();
- public AliyunOSSInputStream(Configuration conf,
- ExecutorService readAheadExecutorService, int maxReadAheadPartNumber,
- AliyunOSSFileSystemStore store, String key, Long contentLength,
- Statistics statistics) throws IOException {
+ public AliyunOSSInputStream(
+ long downloadPartSize,
+ ExecutorService readAheadExecutorService,
+ int maxReadAheadPartNumber,
+ AliyunOSSFileSystemStore store,
+ String key,
+ Long contentLength,
+ Statistics statistics) throws IOException {
this.readAheadExecutorService =
-MoreExecutors.listeningDecorator(readAheadExecutorService);
+MoreExecutors.listeningDecorator(readAheadExecutorService);
this.store = store;
this.key = key;
this.statistics = statistics;
this.contentLength = contentLength;
-downloadPartSize = conf.getLong(MULTIPART_DOWNLOAD_SIZE_KEY,
-MULTIPART_DOWNLOAD_SIZE_DEFAULT);
+this.downloadPartSize = downloadPartSize;
Review Comment:
Good point. I'd like use IO_FILE_BUFFER_SIZE_DEFAULT(4KB) as its min size,
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
