[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=621249&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-621249 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 10/Jul/21 09:40 Start Date: 10/Jul/21 09:40 Worklog Time Spent: 10m Work Description: surendralilhore merged pull request #3106: URL: https://github.com/apache/hadoop/pull/3106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 621249) Time Spent: 4.5h (was: 4h 20m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=621248&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-621248 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 10/Jul/21 09:39 Start Date: 10/Jul/21 09:39 Worklog Time Spent: 10m Work Description: surendralilhore commented on pull request #3106: URL: https://github.com/apache/hadoop/pull/3106#issuecomment-877606783 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 621248) Time Spent: 4h 20m (was: 4h 10m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[
https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=611261&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611261
]
ASF GitHub Bot logged work on HADOOP-17596:
---
Author: ASF GitHub Bot
Created on: 15/Jun/21 11:24
Start Date: 15/Jun/21 11:24
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3106:
URL: https://github.com/apache/hadoop/pull/3106#issuecomment-861416575
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| -1 :x: | mvninstall | 30m 30s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3106/1/artifact/out/branch-mvninstall-root.txt)
| root in branch-3.3 failed. |
| +1 :green_heart: | compile | 0m 37s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 40s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 3s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 16m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 73m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3106/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3106 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 9bd7642b85c1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / e888bb3e01e0695dd02f08986f478897f371414b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3106/1/testReport/ |
| Max. process+thread count | 617 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3106/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 611261)
Time Spent: 4h 10m (was: 4h)
> ABFS: Change default Readahead Queue Depth from num(processors) to const
>
>
> Key: HADOOP-17596
> URL: https://issues.apache.org/jira/browse/HADOOP-17596
> Project: Hadoop Common
>
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=611228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611228 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 15/Jun/21 10:09 Start Date: 15/Jun/21 10:09 Worklog Time Spent: 10m Work Description: sumangala-patki opened a new pull request #3106: URL: https://github.com/apache/hadoop/pull/3106 Contributed by Sumangala Patki. (cherry picked from commit 76d92eb2a22c71b5fcde88a9b4d2faec81a1cb9f) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611228) Time Spent: 4h (was: 3h 50m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=608930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-608930 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 09/Jun/21 05:36 Start Date: 09/Jun/21 05:36 Worklog Time Spent: 10m Work Description: sumangala-patki commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-857390775 @surendralilhore thanks for updating the JIRA @steveloughran yes, but planning to ensure a gap (maybe a week or two) between check-in to trunk and backport of PRs. Will keep a track of pending backports -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 608930) Time Spent: 3h 50m (was: 3h 40m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=608923&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-608923 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 09/Jun/21 05:11 Start Date: 09/Jun/21 05:11 Worklog Time Spent: 10m Work Description: surendralilhore commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-857378838 @steveloughran , I updated the Jira. > Are there any plans to backport to branch-3.3? A retest of the cherrypick is all which should be needed Yes, @sumangala-patki is working on backport and testing it for 3.2 and 3.3 branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 608923) Time Spent: 3h 40m (was: 3.5h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=608720&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-608720 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 08/Jun/21 20:31 Start Date: 08/Jun/21 20:31 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-857108374 1. Now this has been merged, the JIRA MUST be updated with fix version 2. Are there any plans to backport to branch-3.3? A retest of the cherrypick is all which should be needed Keeping both branches in sync is essential for cherrypicking future work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 608720) Time Spent: 3.5h (was: 3h 20m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=606812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-606812 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 04/Jun/21 08:26 Start Date: 04/Jun/21 08:26 Worklog Time Spent: 10m Work Description: surendralilhore commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730 Thanks @anoopsjohn for test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 606812) Time Spent: 3h 20m (was: 3h 10m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=606584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-606584 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 04/Jun/21 08:00 Start Date: 04/Jun/21 08:00 Worklog Time Spent: 10m Work Description: surendralilhore merged pull request #2795: URL: https://github.com/apache/hadoop/pull/2795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 606584) Time Spent: 3h 10m (was: 3h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=606452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-606452 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 04/Jun/21 07:44 Start Date: 04/Jun/21 07:44 Worklog Time Spent: 10m Work Description: surendralilhore edited a comment on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730 Thanks @anoopsjohn for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 606452) Time Spent: 3h (was: 2h 50m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=605784&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605784 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 03/Jun/21 08:56 Start Date: 03/Jun/21 08:56 Worklog Time Spent: 10m Work Description: surendralilhore merged pull request #2795: URL: https://github.com/apache/hadoop/pull/2795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605784) Time Spent: 2h 50m (was: 2h 40m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=605780&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605780 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 03/Jun/21 08:47 Start Date: 03/Jun/21 08:47 Worklog Time Spent: 10m Work Description: surendralilhore commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730 Thanks @anoopsjohn for test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605780) Time Spent: 2.5h (was: 2h 20m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=605781&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605781 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 03/Jun/21 08:47 Start Date: 03/Jun/21 08:47 Worklog Time Spent: 10m Work Description: surendralilhore edited a comment on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730 Thanks @anoopsjohn for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605781) Time Spent: 2h 40m (was: 2.5h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=598127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-598127 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 17/May/21 18:35 Start Date: 17/May/21 18:35 Worklog Time Spent: 10m Work Description: anoopsjohn commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842063424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 598127) Time Spent: 2h 20m (was: 2h 10m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=598094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-598094 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 17/May/21 18:31 Start Date: 17/May/21 18:31 Worklog Time Spent: 10m Work Description: vinaysbadami commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842000750 @anoopsjohn ==> wrt your comment on "will this hamper spark jobs". 1. This is a speculative read ahead. So whenever this is wrong, it is waste io and iops. 2. With parquet etc, the reads tend to be random. Hence a smaller read ahead depth is prefereable 3. Based on debugging various customer perf isues, we rarely saw benefit of > 2 4. This config is per inputstream and not global across streams. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 598094) Time Spent: 2h 10m (was: 2h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=597482&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597482 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 17/May/21 07:04 Start Date: 17/May/21 07:04 Worklog Time Spent: 10m Work Description: anoopsjohn commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842063951 > This will allow passing per input stream options. But this config support is not added yet. We can follow up in a jira. Will be nice to have as a feature support from Driver side. +1 for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597482) Time Spent: 2h (was: 1h 50m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=597481&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597481 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 17/May/21 07:03 Start Date: 17/May/21 07:03 Worklog Time Spent: 10m Work Description: anoopsjohn commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842063424 Thanks @vinaysbadami . Ya speculative part I agree. Points #2 and #3 are very important. You can think whether can include this in abfs.md even. Will be good. >This config is per inputstream and not global across streams. No I dont think so. This config is cluster level and all input streams opened in this will use this value. Now we have openFile(Path path) API usage in AzureBlobFileSystem. This will allow passing per input stream options. But this config support is not added yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597481) Time Spent: 1h 50m (was: 1h 40m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=597460&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597460 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 17/May/21 05:33 Start Date: 17/May/21 05:33 Worklog Time Spent: 10m Work Description: vinaysbadami commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842000750 @anoopsjohn ==> wrt your comment on "will this hamper spark jobs". 1. This is a speculative read ahead. So whenever this is wrong, it is waste io and iops. 2. With parquet etc, the reads tend to be random. Hence a smaller read ahead depth is prefereable 3. Based on debugging various customer perf isues, we rarely saw benefit of > 2 4. This config is per inputstream and not global across streams. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597460) Time Spent: 1h 40m (was: 1.5h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[
https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=596052&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596052
]
ASF GitHub Bot logged work on HADOOP-17596:
---
Author: ASF GitHub Bot
Created on: 13/May/21 11:26
Start Date: 13/May/21 11:26
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-840495522
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 3s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 21s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 75m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 10f225a42410 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 56635ba7beb34ed35ecdf4b88a9065dc12c57af0 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/testReport/ |
| Max. process+thread count | 689 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: h
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[
https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=596016&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596016
]
ASF GitHub Bot logged work on HADOOP-17596:
---
Author: ASF GitHub Bot
Created on: 13/May/21 10:04
Start Date: 13/May/21 10:04
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2795:
URL: https://github.com/apache/hadoop/pull/2795#discussion_r631702871
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -569,6 +570,20 @@ public void testDiffReadRequestSizeAndRAHBlockSize()
throws Exception {
testReadAheads(inputStream, FORTY_EIGHT_KB, SIXTEEN_KB);
}
+ @Test
+ public void testDefaultReadaheadQueueDepth() throws Exception {
+Configuration config = getRawConfiguration();
+config.unset("fs.azure.readaheadqueue.depth");
Review comment:
For each test, a new raw Configuration instance is created and used to
initialize abfsConfig and fileSystem. Hence, changes made to the config within
a test will not affect other tests. This modification will only be reflected in
the fs instance created in getFileSystem(config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 596016)
Time Spent: 1h 20m (was: 1h 10m)
> ABFS: Change default Readahead Queue Depth from num(processors) to const
>
>
> Key: HADOOP-17596
> URL: https://issues.apache.org/jira/browse/HADOOP-17596
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The default value of readahead queue depth is currently set to the number of
> available processors. However, this can result in one inputstream instance
> consuming more processor time. To ensure equal thread allocation during read
> for all inputstreams created in a session, we change the default readahead
> queue depth to a constant (2).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=595922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-595922 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 13/May/21 06:55 Start Date: 13/May/21 06:55 Worklog Time Spent: 10m Work Description: anoopsjohn commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-840357499 When there are not many parallel FIS reads, this (possible) reduction in read ahead parallelism will affect the read throughput? Specially Spark kind of? From HBase standpoint this is acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 595922) Time Spent: 1h 10m (was: 1h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[
https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=595919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-595919
]
ASF GitHub Bot logged work on HADOOP-17596:
---
Author: ASF GitHub Bot
Created on: 13/May/21 06:51
Start Date: 13/May/21 06:51
Worklog Time Spent: 10m
Work Description: anoopsjohn commented on a change in pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#discussion_r631605580
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -569,6 +570,20 @@ public void testDiffReadRequestSizeAndRAHBlockSize()
throws Exception {
testReadAheads(inputStream, FORTY_EIGHT_KB, SIXTEEN_KB);
}
+ @Test
+ public void testDefaultReadaheadQueueDepth() throws Exception {
+Configuration config = getRawConfiguration();
+config.unset("fs.azure.readaheadqueue.depth");
Review comment:
This in effect change this config (Is set already) for this test class.
As we direct change in Raw config.
Should we create a cloned conf from rawConfig and on that unset and use that
for FS created for this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 595919)
Time Spent: 1h (was: 50m)
> ABFS: Change default Readahead Queue Depth from num(processors) to const
>
>
> Key: HADOOP-17596
> URL: https://issues.apache.org/jira/browse/HADOOP-17596
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The default value of readahead queue depth is currently set to the number of
> available processors. However, this can result in one inputstream instance
> consuming more processor time. To ensure equal thread allocation during read
> for all inputstreams created in a session, we change the default readahead
> queue depth to a constant (2).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[
https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=572526&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572526
]
ASF GitHub Bot logged work on HADOOP-17596:
---
Author: ASF GitHub Bot
Created on: 26/Mar/21 08:23
Start Date: 26/Mar/21 08:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-808031395
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 15s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 34s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 87m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 1a2a6de7b3dd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b8fc3459e10b21826844ba99302aa13e5559d71c |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: h
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=570266&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-570266 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 23/Mar/21 06:35 Start Date: 23/Mar/21 06:35 Worklog Time Spent: 10m Work Description: sumangala-patki commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-804657103 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` HNS OAuth [INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 497, Failures: 0, Errors: 0, Skipped: 70 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 49 HNS-SharedKey [INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 495, Failures: 0, Errors: 0, Skipped: 24 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40 Non-HNS SharedKey [INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 250 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 570266) Time Spent: 40m (was: 0.5h) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[
https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=570249&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-570249
]
ASF GitHub Bot logged work on HADOOP-17596:
---
Author: ASF GitHub Bot
Created on: 23/Mar/21 06:05
Start Date: 23/Mar/21 06:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-804643431
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 33s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 51s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 7s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 75m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 75564054a6e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b165323da580b2b1ed88b9c09b9fb42d7891c6f7 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/testReport/ |
| Max. process+thread count | 665 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multib
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=569661&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-569661 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 22/Mar/21 12:07 Start Date: 22/Mar/21 12:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2795: URL: https://github.com/apache/hadoop/pull/2795#issuecomment-804012337 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 17s | | https://github.com/apache/hadoop/pull/2795 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/2795 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 569661) Time Spent: 20m (was: 10m) > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?focusedWorklogId=569659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-569659 ] ASF GitHub Bot logged work on HADOOP-17596: --- Author: ASF GitHub Bot Created on: 22/Mar/21 12:04 Start Date: 22/Mar/21 12:04 Worklog Time Spent: 10m Work Description: sumangala-patki opened a new pull request #2795: URL: https://github.com/apache/hadoop/pull/2795 The default value of readahead queue depth is currently set to the number of available processors. However, this can result in one inputstream instance consuming more processor time. To ensure equal thread allocation during read for all inputstreams created in a session, we change the default readahead queue depth to a constant (2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 569659) Remaining Estimate: 0h Time Spent: 10m > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
