[ https://issues.apache.org/jira/browse/HADOOP-18177?focusedWorklogId=760945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760945 ]
ASF GitHub Bot logged work on HADOOP-18177: ------------------------------------------- Author: ASF GitHub Bot Created on: 22/Apr/22 16:58 Start Date: 22/Apr/22 16:58 Worklog Time Spent: 10m Work Description: steveloughran commented on code in PR #4205: URL: https://github.com/apache/hadoop/pull/4205#discussion_r856383998 ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ########## @@ -0,0 +1,151 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property |Meaning |Default | +|--- |--- |--- | +|fs.s3a.prefetch.enabled |Enable the prefetch input stream |TRUE | Review Comment: 1. use backticks around the configuration names and values; all values must be valid if passed in as the config strings. they will be. 2. use `true` for true, rather than the capitalised value ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ########## @@ -0,0 +1,151 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + Review Comment: can you replace "we" in the docs to refer instead to the class/component which is doing the work ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ########## @@ -0,0 +1,151 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property |Meaning |Default | +|--- |--- |--- | +|fs.s3a.prefetch.enabled |Enable the prefetch input stream |TRUE | +|fs.s3a.prefetch.block.size |Size of a block |8MB | +|fs.s3a.prefetch.block.count |Number of blocks to prefetch |8 | + +### Key Components: + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of this class as the input stream. Depending on the file size, it will either use the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the file size < configured block size. Will read the entire file into memory. + +`S3CachingInputStream` - Underlying input stream used when file size > configured block size. Uses asynchronous prefetching of blocks and caching to improve performance. + +`BlockData` - Holds information about the blocks in a file, such as: + +* Number of blocks in the file +* Block size +* State of each block (initially all blocks have state *NOT_READY*). Other states are: Queued, Ready, Cached. + +`BufferData` - Holds the buffer and additional information about it such as: + +* The block number this buffer is for +* State of the buffer (Unknown, Blank, Prefetching, Caching, Ready, Done). Initial state of a buffer is blank. + +`CachingBlockManager` - Implements reading data into the buffer, prefetching and caching. + +`BufferPool` - Manages a fixed sized pool of buffers. It’s used by `CachingBlockManager` to acquire buffers. + +`S3File` - Implements operations to interact with S3 such as opening and closing the input stream to the S3 file. + +`S3Reader` - Implements reading from the stream opened by `S3File`. Reads from this input stream in blocks of 64KB. + +`FilePosition` - Provides functionality related to tracking the position in the file. Also gives access to the current buffer in use. + +`SingleFilePerBlockCache` - Responsible for caching blocks to the local file system. Each cache block is stored on the local disk as a separate file. + +### Operation + +### S3InMemoryInputStream + +If we have a file with size 5MB, and block size = 8MB. Since file size is less than the block size, the `S3InMemoryInputStream` will be used. + +If the caller makes the following read calls: + + +``` +in.read(buffer, 0, 3MB); +in.read(buffer, 0, 2MB); +``` + +When the first read is issued, there is no buffer in use yet. We get the data in this file by calling the `ensureCurrentBuffer()` method, which ensures that a buffer with data is available to be read from. + +The `ensureCurrentBuffer()` then: + +* Reads data into a buffer by calling `S3Reader.read(ByteBuffer buffer, long offset, int size)` +* `S3Reader` uses `S3File` to open an input stream to the S3 file by making a `getObject()` request with range as `(0, filesize)`. +* The S3Reader reads the entire file into the provided buffer, and once reading is complete closes the S3 stream and frees all underlying resources. +* Now the entire file is in a buffer, set this data in `FilePosition` so it can be accessed by the input stream. + +The read operation now just gets the required bytes from the buffer in `FilePosition`. + +When the second read is issued, there is already a valid buffer which can be used. Don’t do anything else, just read the required bytes from this buffer. + +### S3CachingInputStream + + + +[Image: image.png] Review Comment: they go into hadoop-tools/hadoop-aws/src/site/resources/images ; other modules have examples of this ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ########## @@ -0,0 +1,151 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property |Meaning |Default | +|--- |--- |--- | +|fs.s3a.prefetch.enabled |Enable the prefetch input stream |TRUE | +|fs.s3a.prefetch.block.size |Size of a block |8MB | +|fs.s3a.prefetch.block.count |Number of blocks to prefetch |8 | + +### Key Components: + +`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of this class as the input stream. Depending on the file size, it will either use the `S3InMemoryInputStream` or the `S3CachingInputStream` as the underlying input stream. + +`S3InMemoryInputStream` - Underlying input stream used when the file size < configured block size. Will read the entire file into memory. + +`S3CachingInputStream` - Underlying input stream used when file size > configured block size. Uses asynchronous prefetching of blocks and caching to improve performance. + +`BlockData` - Holds information about the blocks in a file, such as: + +* Number of blocks in the file +* Block size +* State of each block (initially all blocks have state *NOT_READY*). Other states are: Queued, Ready, Cached. + +`BufferData` - Holds the buffer and additional information about it such as: + +* The block number this buffer is for +* State of the buffer (Unknown, Blank, Prefetching, Caching, Ready, Done). Initial state of a buffer is blank. + +`CachingBlockManager` - Implements reading data into the buffer, prefetching and caching. + +`BufferPool` - Manages a fixed sized pool of buffers. It’s used by `CachingBlockManager` to acquire buffers. + +`S3File` - Implements operations to interact with S3 such as opening and closing the input stream to the S3 file. + +`S3Reader` - Implements reading from the stream opened by `S3File`. Reads from this input stream in blocks of 64KB. + +`FilePosition` - Provides functionality related to tracking the position in the file. Also gives access to the current buffer in use. + +`SingleFilePerBlockCache` - Responsible for caching blocks to the local file system. Each cache block is stored on the local disk as a separate file. + +### Operation + +### S3InMemoryInputStream + +If we have a file with size 5MB, and block size = 8MB. Since file size is less than the block size, the `S3InMemoryInputStream` will be used. + +If the caller makes the following read calls: + + +``` +in.read(buffer, 0, 3MB); +in.read(buffer, 0, 2MB); +``` + +When the first read is issued, there is no buffer in use yet. We get the data in this file by calling the `ensureCurrentBuffer()` method, which ensures that a buffer with data is available to be read from. + +The `ensureCurrentBuffer()` then: + +* Reads data into a buffer by calling `S3Reader.read(ByteBuffer buffer, long offset, int size)` +* `S3Reader` uses `S3File` to open an input stream to the S3 file by making a `getObject()` request with range as `(0, filesize)`. +* The S3Reader reads the entire file into the provided buffer, and once reading is complete closes the S3 stream and frees all underlying resources. +* Now the entire file is in a buffer, set this data in `FilePosition` so it can be accessed by the input stream. + +The read operation now just gets the required bytes from the buffer in `FilePosition`. + +When the second read is issued, there is already a valid buffer which can be used. Don’t do anything else, just read the required bytes from this buffer. + +### S3CachingInputStream + + + +[Image: image.png] + +Now, if we have a file with size 40MB, and block size = 8MB. The `S3CachingInputStream` will be used. + +### Sequential Reads + +If the caller makes the following calls: + +``` +in.read(buffer, 0, 5MB) +in.read(buffer, 0, 8MB) +``` + +For the first read call, there is no valid buffer yet. `ensureCurrentBuffer()` is called, and for the first read(), prefetch count is set as 1. Review Comment: add backticks around 'read()' ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ########## @@ -0,0 +1,151 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. Review Comment: should the doc use Object to refer to something in s3, to distinguish it from the local FS files? ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md: ########## @@ -0,0 +1,151 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3A Prefetching + + +This document explains the `S3PrefetchingInputStream` and the various components it uses. + +This input stream implements prefetching and caching to improve read performance of the input stream. A high level overview of this feature can also be found on [this](https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0) blogpost. + +With prefetching, we divide the file into blocks of a fixed size (default is 8MB), associate buffers to these blocks, and then read data into these buffers asynchronously. We also potentially cache these blocks. + +### Basic Concepts + +* **File** : A binary blob of data stored on some storage device. +* **Block :** A file is divided into a number of blocks. The default size of a block is 8MB, but can be configured. The size of the first n-1 blocks is same, and the size of the last block may be same or smaller. +* **Block based reading** : The granularity of read is one block. That is, we read an entire block and return or none at all. Multiple blocks may be read in parallel. + +### Configuring the stream + +|Property |Meaning |Default | +|--- |--- |--- | +|fs.s3a.prefetch.enabled |Enable the prefetch input stream |TRUE | +|fs.s3a.prefetch.block.size |Size of a block |8MB | Review Comment: 8MB isn't gong to be a valid value. `8M` should be Issue Time Tracking ------------------- Worklog Id: (was: 760945) Time Spent: 50m (was: 40m) > document use and architecture design of prefetching s3a input stream > -------------------------------------------------------------------- > > Key: HADOOP-18177 > URL: https://issues.apache.org/jira/browse/HADOOP-18177 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 > Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Ahmar Suhail > Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Document S3PrefetchingInputStream for users (including any new failure modes > in troubleshooting) and the architecture for maintainers > there's some markdown in > hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md > already -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org