[ https://issues.apache.org/jira/browse/HADOOP-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824625#comment-17824625 ]
ASF GitHub Bot commented on HADOOP-19102: ----------------------------------------- saxenapranav opened a new pull request, #6617: URL: https://github.com/apache/hadoop/pull/6617 JIRA: https://issues.apache.org/jira/browse/HADOOP-19102 The method `optimisedRead` creates a buffer array of size `readBufferSize`. If footerReadBufferSize is greater than readBufferSize, abfs will attempt to read more data than the buffer array can hold, which causes an exception. Change: To avoid this, we will keep footerBufferSize = min(readBufferSizeConfig, footerBufferSizeConfig) Also, as part of this PR, have improved tests within `ITestAbfsInputStreamReadFooter`. There are tests which have multiple combination, and there was file getting created for all the combination. There has to be a combination on different fileSize. The change: We will spin up one thread each for each fileSize. And in each thread, all the combination for that particular fileSize will run. This will help in creating file once for a fileSize and multiple fileSize related assertion can happen in parallel and use hardware capability. Improvement: on a 6 processor VM, on trunk, it tool 8min47sec to run all tests of ITestAbfsInputStreamReadFooter and in the PR branch, it took 7 min. (Each test method run one after another). > [ABFS]: FooterReadBufferSize should not be greater than readBufferSize > ---------------------------------------------------------------------- > > Key: HADOOP-19102 > URL: https://issues.apache.org/jira/browse/HADOOP-19102 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Reporter: Pranav Saxena > Assignee: Pranav Saxena > Priority: Major > Fix For: 3.4.0, 3.5.0 > > > The method `optimisedRead` creates a buffer array of size `readBufferSize`. > If footerReadBufferSize is greater than readBufferSize, abfs will attempt to > read more data than the buffer array can hold, which causes an exception. > Change: To avoid this, we will keep footerBufferSize = > min(readBufferSizeConfig, footerBufferSizeConfig) > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org