[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=759124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759124 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 20/Apr/22 11:10 Start Date: 20/Apr/22 11:10 Worklog Time Spent: 10m Work Description: ahmarsuhail opened a new pull request, #4205: URL: https://github.com/apache/hadoop/pull/4205 ### Description of PR Documents usage and architecture of the prefetching input stream. Issue Time Tracking --- Worklog Id: (was: 759124) Remaining Estimate: 0h Time Spent: 10m > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=759187&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759187 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 20/Apr/22 12:45 Start Date: 20/Apr/22 12:45 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1103889770 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 52s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 41m 30s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 56s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 18s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 39s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/1/artifact/out/blanks-tabs.txt) | The patch 5 line(s) with tabs. | | +1 :green_heart: | mvnsite | 0m 39s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 31s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | -1 :x: | asflicense | 0m 44s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. | | | | 93m 51s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux 1c8ff3f57b1d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / b089cf68fb46f7fdbb5d1d88b1b7d87d3988ebfc | | Max. process+thread count | 601 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/1/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 759187) Time Spent: 20m (was: 10m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=759259&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759259 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 20/Apr/22 14:25 Start Date: 20/Apr/22 14:25 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1103997065 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 49s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 42m 8s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 55s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 19s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 38s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/2/artifact/out/blanks-tabs.txt) | The patch 1 line(s) with tabs. | | +1 :green_heart: | mvnsite | 0m 40s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. | | | | 93m 33s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux dfc2242ac9ef 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / e5e9ea3075349df41fb52b5ae48b9f9cc3ed2d78 | | Max. process+thread count | 522 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/2/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 759259) Time Spent: 0.5h (was: 20m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=759851&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759851 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 21/Apr/22 09:04 Start Date: 21/Apr/22 09:04 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r854934280 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. Review Comment: Let's put the title in the link somehow for screenreaders. ```suggestion This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature was published in [Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0). ``` ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. Review Comment: colon is inside the bold here but not for others ```suggestion * **File**: A binary blob of data stored on some storage device. * **Block**: A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. * **Block based reading**: The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. ``` ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | +|fs.s3a.prefetch.block.size|Size of a block|8MB| +|fs.s3a.prefetch.block.count|Number of blocks to prefetch|8| + +### Key Components: + +`S3PrefetchingInputStream`
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=760945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760945 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 22/Apr/22 16:58 Start Date: 22/Apr/22 16:58 Worklog Time Spent: 10m Work Description: steveloughran commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r856383998 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | Review Comment: 1. use backticks around the configuration names and values; all values must be valid if passed in as the config strings. they will be. 2. use `true` for true, rather than the capitalised value ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + Review Comment: can you replace "we" in the docs to refer instead to the class/component which is doing the work ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | +|fs.s3a.prefetch.block.size|Size of a block|8MB| +|fs.s3a.prefetch.block.count|Number of blocks to prefetch|8| + +### Key Components: + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of this class as the input stream. Depending on the file size, it will either use the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the file size < configured
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=761912&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761912 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 25/Apr/22 17:00 Start Date: 25/Apr/22 17:00 Worklog Time Spent: 10m Work Description: ahmarsuhail commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r857842762 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. Review Comment: good point, have removed Issue Time Tracking --- Worklog Id: (was: 761912) Time Spent: 1h 10m (was: 1h) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=761910&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761910 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 25/Apr/22 17:00 Start Date: 25/Apr/22 17:00 Worklog Time Spent: 10m Work Description: ahmarsuhail commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r857842464 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,151 @@ + + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | +|fs.s3a.prefetch.block.size|Size of a block|8MB| +|fs.s3a.prefetch.block.count|Number of blocks to prefetch|8| + +### Key Components: + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of this class as the input stream. Depending on the file size, it will either use the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the file size < configured block size. Will read the entire file into memory. + +`S3CachingInputStream` - Underlying input stream used when file size > configured block size. Uses asynchronous prefetching of blocks and caching to improve performance. + +`BlockData` - Holds information about the blocks in a file, such as: + +* Number of blocks in the file +* Block size +* State of each block (initially all blocks have state *NOT_READY*). Other states are: Queued, Ready, Cached. + +`BufferData` - Holds the buffer and additional information about it such as: + +* The block number this buffer is for +* State of the buffer (Unknown, Blank, Prefetching, Caching, Ready, Done). Initial state of a buffer is blank. + +`CachingBlockManager` - Implements reading data into the buffer, prefetching and caching. + +`BufferPool` - Manages a fixed sized pool of buffers. It’s used by `CachingBlockManager` to acquire buffers. + +`S3File` - Implements operations to interact with S3 such as opening and closing the input stream to the S3 file. + +`S3Reader` - Implements reading from the stream opened by `S3File`. Reads from this input stream in blocks of 64KB. + +`FilePosition` - Provides functionality related to tracking the position in the file. Also gives access to the current buffer in use. + +`SingleFilePerBlockCache` - Responsible for caching blocks to the local file system. Each cache block is stored on the local disk as a separate file. + +### Operation + +### S3InMemoryInputStream + +If we have a file with size 5MB, and block size = 8MB. Since file size is less than the block size, the `S3InMemoryInputStream` will be used. + +If the caller makes the following read calls: + + +``` +in.read(buffer, 0, 3MB); +in.read(buffer, 0, 2MB); +``` + +When the first read is issued, there is no buffer in use yet. We get the data in this file by calling the `ensureCurrentBuffer()` method, which ensures that a buffer with data is available to be read from. + +The `ensureCurrentBuffer()` then: + +* Reads data into a buffer by calling `S3Reader.read(ByteBuffer buffer, long offset, int size)` +* `S3Reader` uses `S3File` to open an input stream to the S3 file by making a `getObject()` request with range as `(0, filesize)`. +* The S3Reader reads the entire file into the provided buffer, and once reading is complete closes the S3 stream and frees all underlying resources. +* Now the entire file is in a buffer, set this data in `FilePosition` so it can be accessed by the input stream. + +The read operation now just
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=761957&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761957 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 25/Apr/22 18:23 Start Date: 25/Apr/22 18:23 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1108897168 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 18m 10s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 41m 49s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 55s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 65m 51s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 39s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/3/artifact/out/blanks-tabs.txt) | The patch 1 line(s) with tabs. | | +1 :green_heart: | mvnsite | 0m 39s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 6s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. | | | | 109m 59s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux f50a219ad738 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / 24380d95f1da4aab12af6183f4302e513e87f6dc | | Max. process+thread count | 520 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/3/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 761957) Time Spent: 1h 20m (was: 1h 10m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=761973&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761973 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 25/Apr/22 18:37 Start Date: 25/Apr/22 18:37 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1108910254 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 18m 3s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 41m 46s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 56s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 3s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 38s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/4/artifact/out/blanks-tabs.txt) | The patch 1 line(s) with tabs. | | +1 :green_heart: | mvnsite | 0m 41s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 22s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. | | | | 110m 18s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux ba91b572c6c8 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / 24380d95f1da4aab12af6183f4302e513e87f6dc | | Max. process+thread count | 521 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/4/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 761973) Time Spent: 1.5h (was: 1h 20m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762184&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762184 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 09:04 Start Date: 26/Apr/22 09:04 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r858455675 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) +blogpost. -With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. +With prefetching, the input stream divides the remote file into blocks of a fixed size, associates +buffers to these blocks and then reads data into these buffers asynchronously. It also potentially +caches these blocks. ### Basic Concepts -* **File** : A binary blob of data stored on some storage device. -* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. -* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. +* **Remote File**: A binary blob of data stored on some storage device. +* **Block**: A file is divided into a number of blocks. The size of the first n-1 blocks is same, + and the size of the last block may be same or smaller. +* **Block based reading**: The granularity of read is one block. That is, either an entire block is + read and returned or none at all. Multiple blocks may be read in parallel. ### Configuring the stream |Property|Meaning|Default| |--- |---|---| -|fs.s3a.prefetch.enabled|Enable the prefetch input stream|TRUE | -|fs.s3a.prefetch.block.size|Size of a block|8MB| -|fs.s3a.prefetch.block.count|Number of blocks to prefetch|8| +|fs.s3a.prefetch.enabled|Enable the prefetch input stream|`true` | +|fs.s3a.prefetch.block.size|Size of a block|`8M`| +|fs.s3a.prefetch.block.count|Number of blocks to prefetch|`8`| Review Comment: We want backticks around the configuration keys too. ```suggestion |`fs.s3a.prefetch.enabled`|Enable the prefetch input stream|`true` | |`fs.s3a.prefetch.block.size`|Size of a block|`8M`| |`fs.s3a.prefetch.block.count`|Number of blocks to prefetch|`8`| ``` ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. +This input stream implements prefetching and caching to improve read performance of the input +stream. A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) +blogpost. Review Comment: drop `blogpost` as it is already mentioned earlier ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -14,94 +14,114 @@ # S3A Prefetching - This document explains the `S3PrefetchingInputStream` and the various components it uses. -This input stream implements prefetching
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762219&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762219 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 10:38 Start Date: 26/Apr/22 10:38 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109635269 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 52s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 41m 38s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 56s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 33s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 41s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/artifact/out/blanks-tabs.txt) | The patch 1 line(s) with tabs. | | +1 :green_heart: | mvnsite | 0m 45s | | the patch passed | | +1 :green_heart: | shadedclient | 24m 10s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. | | | | 94m 31s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux 9472cd5389b4 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / ba1d26a0f77b6d3795cdeb923c1403dfa31e7ded | | Max. process+thread count | 518 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/5/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 762219) Time Spent: 1h 50m (was: 1h 40m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762237&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762237 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 11:18 Start Date: 26/Apr/22 11:18 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109670724 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 54s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 42m 23s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 53s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 45s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 39s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/blanks-eol.txt) | The patch has 30 line(s) that end in blanks. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/blanks-tabs.txt) | The patch 1 line(s) with tabs. | | +1 :green_heart: | mvnsite | 0m 40s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. | | | | 94m 11s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux afff6318d18d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / 125152ef64d24083abb7308978302285c1720fb6 | | Max. process+thread count | 589 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/6/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 762237) Time Spent: 2h (was: 1h 50m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762239 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 11:28 Start Date: 26/Apr/22 11:28 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r858597582 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ## @@ -0,0 +1,192 @@ + + +# S3A Prefetching + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input +stream. +A high level overview of this feature was published in +[Pinterest Engineering's blog post titled "Improving efficiency and reducing runtime using S3 read optimization"](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0). + +With prefetching, the input stream divides the remote file into blocks of a fixed size, associates +buffers to these blocks and then reads data into these buffers asynchronously. +It also potentially caches these blocks. + +### Basic Concepts + +* **Remote File**: A binary blob of data stored on some storage device. +* **Block File**: Local file containing a block of the remote file. +* **Block**: A file is divided into a number of blocks. +The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading**: The granularity of read is one block. +That is, either an entire block is read and returned or none at all. +Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property|Meaning|Default| +|--- |---|---| +|`fs.s3a.prefetch.enabled`|Enable the prefetch input stream|`true` | +|`fs.s3a.prefetch.block.size`|Size of a block|`8M`| +|`fs.s3a.prefetch.block.count`|Number of blocks to prefetch|`8`| + +### Key Components + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of +this class as the input stream. +Depending on the remote file size, it will either use +the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the remote file size < configured block +size. +Will read the entire remote file into memory. + +`S3CachingInputStream` - Underlying input stream used when remote file size > configured block size. +Uses asynchronous prefetching of blocks and caching to improve performance. + +`BlockData` - Holds information about the blocks in a remote file, such as: + +* Number of blocks in the remote file +* Block size +* State of each block (initially all blocks have state *NOT_READY*). +Other states are: Queued, Ready, Cached. + +`BufferData` - Holds the buffer and additional information about it such as: + +* The block number this buffer is for +* State of the buffer (Unknown, Blank, Prefetching, Caching, Ready, Done). +Initial state of a buffer is blank. + +`CachingBlockManager` - Implements reading data into the buffer, prefetching and caching. + +`BufferPool` - Manages a fixed sized pool of buffers. +It’s used by `CachingBlockManager` to acquire buffers. + +`S3File` - Implements operations to interact with S3 such as opening and closing the input stream to +the remote file in S3. + +`S3Reader` - Implements reading from the stream opened by `S3File`. +Reads from this input stream in blocks of 64KB. + +`FilePosition` - Provides functionality related to tracking the position in the file. +Also gives access to the current buffer in use. + +`SingleFilePerBlockCache` - Responsible for caching blocks to the local file system. +Each cache block is stored on the local disk as a separate block file. + +### Operation + + S3InMemoryInputStream + +For a remote file with size 5MB, and block size = 8MB, since file size is less than the block size, +the `S3InMemoryInputStream` will be used. + +If the caller makes the following read calls: + +``` +in.read(buffer, 0, 3MB); +in.read(buffer, 0, 2MB); +``` + +When the first read is issued, there is no buffer in use yet. +The `S3InMemoryInputStream` gets the data in this remote file by calling the `ensureCurrentBuffer()` +method, which ensures that a buffer with data is available to be read from. + +The `ensureCurrentBuffer()` then: + +* Reads data into a buffer by calling `S3Reader.read(ByteBuffer buffer, long offset, int size)`. +* `S3Reader` uses `S3File` to open an input stream to the remote file in S3 by making + a `getObject()` request with range as `(0, filesize)`. +* The `S3Reader` reads the entire re
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762317 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 13:49 Start Date: 26/Apr/22 13:49 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109822081 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 50s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 42m 6s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 55s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 12s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 38s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | mvnsite | 0m 40s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 34s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. | | | | 93m 28s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux 918724e3c41c 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / 658396acb384ba50103e240e890f31cf1355388a | | Max. process+thread count | 601 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/7/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 762317) Time Spent: 2h 20m (was: 2h 10m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762347&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762347 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 14:22 Start Date: 26/Apr/22 14:22 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4205: URL: https://github.com/apache/hadoop/pull/4205#issuecomment-1109860466 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 54s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | _ feature-HADOOP-18028-s3a-prefetch Compile Tests _ | | +1 :green_heart: | mvninstall | 41m 59s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | mvnsite | 0m 56s | | feature-HADOOP-18028-s3a-prefetch passed | | +1 :green_heart: | shadedclient | 66m 18s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 38s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | mvnsite | 0m 41s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 32s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. | | | | 93m 36s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4205 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint | | uname | Linux 416a76a29726 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | feature-HADOOP-18028-s3a-prefetch / 9558361263d879b0bef5f452528dca5f18abdc15 | | Max. process+thread count | 525 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4205/8/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. Issue Time Tracking --- Worklog Id: (was: 762347) Time Spent: 2.5h (was: 2h 20m) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18177) document use and architecture design of prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=762448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762448 ] ASF GitHub Bot logged work on HADOOP-18177: --- Author: ASF GitHub Bot Created on: 26/Apr/22 17:36 Start Date: 26/Apr/22 17:36 Worklog Time Spent: 10m Work Description: steveloughran merged PR #4205: URL: https://github.com/apache/hadoop/pull/4205 Issue Time Tracking --- Worklog Id: (was: 762448) Time Spent: 2h 40m (was: 2.5h) > document use and architecture design of prefetching s3a input stream > > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org