[
https://issues.apache.org/jira/browse/HADOOP-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825921#comment-17825921
]
ASF GitHub Bot commented on HADOOP-19102:
-----------------------------------------
saxenapranav commented on code in PR #6617:
URL: https://github.com/apache/hadoop/pull/6617#discussion_r1522594974
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java:
##########
@@ -54,9 +63,44 @@ public class ITestAbfsInputStreamReadFooter extends
ITestAbfsInputStream {
private static final int TEN = 10;
private static final int TWENTY = 20;
+ private static ExecutorService executorService;
+
+ private static final int SIZE_256_KB = 256 * ONE_KB;
+
+ private static final Integer[] FILE_SIZES = {
Review Comment:
On trunk, file size has the range: 256KB, 512KB, 1MB, 2MB, 4MB.
As part of this PR, fileSize has the range 256KB, 512KB, 1MB, 4MB. And as
part of this PR, a dimension of readBufferSize is added [256 KB, 512KB, 1MB,
4MB]. With this PR. for a test, at a given fileSize, only once the file would
be created, and all the combinations of readBufferSize and footerReadBufferSize
would test on that file.
On trunk, if we run all the test sequentially, it takes ~8min47sec and if
all tests are run on this PR (including readBufferSize dimension), it takes
only ~7min (These tests runs are done out of Azure network). With this PR, the
time to run this class reduces.
4MB fileSize is included because we have a default readBufferSize of 4MB.
Kindly advise please if we should remove the 4MB fileSize from the fileSizes.
Thank you!
> [ABFS]: FooterReadBufferSize should not be greater than readBufferSize
> ----------------------------------------------------------------------
>
> Key: HADOOP-19102
> URL: https://issues.apache.org/jira/browse/HADOOP-19102
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0
> Reporter: Pranav Saxena
> Assignee: Pranav Saxena
> Priority: Major
> Labels: pull-request-available
>
> The method `optimisedRead` creates a buffer array of size `readBufferSize`.
> If footerReadBufferSize is greater than readBufferSize, abfs will attempt to
> read more data than the buffer array can hold, which causes an exception.
> Change: To avoid this, we will keep footerBufferSize =
> min(readBufferSizeConfig, footerBufferSizeConfig)
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
