[jira] [Commented] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[ https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038675#comment-17038675 ] Hudson commented on HADOOP-16759: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17961 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17961/]) HADOOP-16759. FileSystem Javadocs to list what breaks on API changes (github: rev a562942b05a29d29e4a4fe3df4f35e2a8e7d208d) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java > Filesystem openFile() builder to take a FileStatus param > > > Key: HADOOP-16759 > URL: https://issues.apache.org/jira/browse/HADOOP-16759 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.0 > > > Let us pass in a file status to openFile() so that S3A & ABFS will skip > their own HEAD requests just to see if a file is there, a normal file and get > its length + etag, > {code} > CompletableFuture streamF = fs.openFile(stat.getPath()) > .withFileStatus(stat).build() > {code} > code opening files off a listing of everything in a directory can eliminate a > lot of requests here. > Also: change the specification of openFile's completable future to say > "returned stream may only raise FNFE, access restrictions on the first read" > That is: it's not just potentially an async open, it's possibly lazy > evaluated entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[ https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021329#comment-17021329 ] Mingliang Liu commented on HADOOP-16759: That makes sense. Thanks! > Filesystem openFile() builder to take a FileStatus param > > > Key: HADOOP-16759 > URL: https://issues.apache.org/jira/browse/HADOOP-16759 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.0 > > > Let us pass in a file status to openFile() so that S3A & ABFS will skip > their own HEAD requests just to see if a file is there, a normal file and get > its length + etag, > {code} > CompletableFuture streamF = fs.openFile(stat.getPath()) > .withFileStatus(stat).build() > {code} > code opening files off a listing of everything in a directory can eliminate a > lot of requests here. > Also: change the specification of openFile's completable future to say > "returned stream may only raise FNFE, access restrictions on the first read" > That is: it's not just potentially an async open, it's possibly lazy > evaluated entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[ https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021009#comment-17021009 ] Steve Loughran commented on HADOOP-16759: - oh, thanks for the merge! I need to a full hadoop-aws backport to branch-3.2 to get all of this in. Now the API is complete (AFAIK) it makes sense, as we can guarantee that all implementations of openFile have the new builder option > Filesystem openFile() builder to take a FileStatus param > > > Key: HADOOP-16759 > URL: https://issues.apache.org/jira/browse/HADOOP-16759 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.0 > > > Let us pass in a file status to openFile() so that S3A & ABFS will skip > their own HEAD requests just to see if a file is there, a normal file and get > its length + etag, > {code} > CompletableFuture streamF = fs.openFile(stat.getPath()) > .withFileStatus(stat).build() > {code} > code opening files off a listing of everything in a directory can eliminate a > lot of requests here. > Also: change the specification of openFile's completable future to say > "returned stream may only raise FNFE, access restrictions on the first read" > That is: it's not just potentially an async open, it's possibly lazy > evaluated entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[ https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020618#comment-17020618 ] Hudson commented on HADOOP-16759: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17888 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17888/]) HADOOP-16759. Filesystem openFile() builder to take a FileStatus param (liuml07: rev 5e2ce370a322a46b496541ccd17443197fcfeb5a) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FutureDataInputStreamBuilder.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3Select.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureDataInputStreamBuilderImpl.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractOpenTest.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstreambuilder.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java > Filesystem openFile() builder to take a FileStatus param > > > Key: HADOOP-16759 > URL: https://issues.apache.org/jira/browse/HADOOP-16759 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Let us pass in a file status to openFile() so that S3A & ABFS will skip > their own HEAD requests just to see if a file is there, a normal file and get > its length + etag, > {code} > CompletableFuture streamF = fs.openFile(stat.getPath()) > .withFileStatus(stat).build() > {code} > code opening files off a listing of everything in a directory can eliminate a > lot of requests here. > Also: change the specification of openFile's completable future to say > "returned stream may only raise FNFE, access restrictions on the first read" > That is: it's not just potentially an async open, it's possibly lazy > evaluated entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[ https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994627#comment-16994627 ] Steve Loughran commented on HADOOP-16759: - adding a new withFileStatus() Builder option is straightforward; it will be something else to pass down to FS/FC implementations, which will change the signature of implementations of the (unshipped) method openFileWithOptions. * add this before any back ports * propose changing openFileWithOptions to take a new OpenFileOptions class for this and future extensibility. If we extend the builder again, we would be just add new fields to OpenFileOptions > Filesystem openFile() builder to take a FileStatus param > > > Key: HADOOP-16759 > URL: https://issues.apache.org/jira/browse/HADOOP-16759 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Let us pass in a file status to openFile() so that S3A & ABFS will skip > their own HEAD requests just to see if a file is there, a normal file and get > its length + etag, > {code} > CompletableFuture streamF = fs.openFile(stat.getPath()) > .withFileStatus(stat).build() > {code} > code opening files off a listing of everything in a directory can eliminate a > lot of requests here. > Also: change the specification of openFile's completable future to say > "returned stream may only raise FNFE, access restrictions on the first read" > That is: it's not just potentially an async open, it's possibly lazy > evaluated entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org