[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805105020 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BlockOperations.java ## @@ -0,0 +1,417 @@ +/* + * Licensed to the Apa

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805104460 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BlockData.java ## @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache So

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805104237 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/SingleFilePerBlockCache.java ## @@ -0,0 +1,328 @@ +/* + * Licensed to

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805104105 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/SingleFilePerBlockCache.java ## @@ -0,0 +1,328 @@ +/* + * Licensed to

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805103846 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AUnbuffer.java ## @@ -67,10 +69,10 @@ public void testUnbuffer() t

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805103066 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AUnbuffer.java ## @@ -67,10 +69,10 @@ public void testUnbuffer() t

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805102478 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java ## @@ -276,6 +279,19 @@ private TransferManager tr

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r805102332 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3EInputStream.java ## @@ -0,0 +1,240 @@ +/* + * Licensed to the Ap

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778488178 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3Reader.java ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache S

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778487448 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3InputStream.java ## @@ -0,0 +1,459 @@ +/* + * Licensed to the Apa

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778486837 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3InputStream.java ## @@ -0,0 +1,459 @@ +/* + * Licensed to the Apa

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778486303 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3InputStream.java ## @@ -0,0 +1,459 @@ +/* + * Licensed to the Apa

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778485852 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java ## @@ -0,0 +1,180 @@ +/* + * Licensed to

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778475353 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Sof

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778475249 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/read/S3FileTest.java ## @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r778456947 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestObjectChangeDetectionAttributes.java ## @@ -234,6 +234,7 @@

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762361371 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -540,7 +540,7 @@ private Constants() { public

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762361014 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java ## @@ -3926,12 +3985,12 @@ public void copyFromLocalF

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762360921 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BlockData.java ## @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache So

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762354915 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/Validate.java ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Sof

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762350797 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BlockData.java ## @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache So

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762192697 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BlockData.java ## @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache So

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r762190855 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BlockCache.java ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache So

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r761389560 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BoundedResourcePool.java ## @@ -0,0 +1,179 @@ +/* + * Licensed to the

[GitHub] [hadoop] bhalchandrap commented on a change in pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

bhalchandrap commented on a change in pull request #3736: URL: https://github.com/apache/hadoop/pull/3736#discussion_r761368184 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BoundedResourcePool.java ## @@ -0,0 +1,179 @@ +/* + * Licensed to the