[jira] [Updated] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
[
https://issues.apache.org/jira/browse/HDFS-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16105:
--
Labels: pull-request-available (was: )
> Edit log corruption due to mismatch between fileId and path
> ---
>
> Key: HDFS-16105
> URL: https://issues.apache.org/jira/browse/HDFS-16105
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.2.2, 3.3.1
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We do stress testing in FUSE with HDFS, Standby Namenode crashes.
> The log is following
> {quote}
> 2021-06-25 17:13:02,953 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
> on operation AddBlockOp [path=/xxx/fiotest_write.354.46,
> penultimateBlock=xxx, lastBlock=xxx, numOfBytes=0}, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File /xxx/fiotest_write.354.46 does not exist.
> {quote}
> The following steps can reproduce it (Illegal writes).
> 1 create file A (fileId X) (1st block is being written)
> 2 rename file A to file B(stile fileId X)
> 3 continue to write file A(using above the same outputstream, 2nd block need
> to be written)
> 4 standby namenode load the above edits and would crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
[
https://issues.apache.org/jira/browse/HDFS-16105?focusedWorklogId=617423=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617423
]
ASF GitHub Bot logged work on HDFS-16105:
-
Author: ASF GitHub Bot
Created on: 01/Jul/21 05:54
Start Date: 01/Jul/21 05:54
Worklog Time Spent: 10m
Work Description: ferhui opened a new pull request #3161:
URL: https://github.com/apache/hadoop/pull/3161
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 617423)
Remaining Estimate: 0h
Time Spent: 10m
> Edit log corruption due to mismatch between fileId and path
> ---
>
> Key: HDFS-16105
> URL: https://issues.apache.org/jira/browse/HDFS-16105
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.2.2, 3.3.1
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We do stress testing in FUSE with HDFS, Standby Namenode crashes.
> The log is following
> {quote}
> 2021-06-25 17:13:02,953 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
> on operation AddBlockOp [path=/xxx/fiotest_write.354.46,
> penultimateBlock=xxx, lastBlock=xxx, numOfBytes=0}, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File /xxx/fiotest_write.354.46 does not exist.
> {quote}
> The following steps can reproduce it (Illegal writes).
> 1 create file A (fileId X) (1st block is being written)
> 2 rename file A to file B(stile fileId X)
> 3 continue to write file A(using above the same outputstream, 2nd block need
> to be written)
> 4 standby namenode load the above edits and would crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
Hui Fei created HDFS-16105:
--
Summary: Edit log corruption due to mismatch between fileId and
path
Key: HDFS-16105
URL: https://issues.apache.org/jira/browse/HDFS-16105
Project: Hadoop HDFS
Issue Type: Bug
Components: namanode
Affects Versions: 3.3.1, 3.2.2
Reporter: Hui Fei
We do stress testing in FUSE with HDFS, Standby Namenode crashes.
The log is following
{quote}
2021-06-25 17:13:02,953 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
on operation AddBlockOp [path=/xxx/fiotest_write.354.46, penultimateBlock=xxx,
lastBlock=xxx, numOfBytes=0}, RpcClientId=, RpcCallId=-2]
java.io.FileNotFoundException: File /xxx/fiotest_write.354.46 does not exist.
{quote}
The following steps can reproduce it (Illegal writes).
1 create file A (fileId X) (1st block is being written)
2 rename file A to file B(stile fileId X)
3 continue to write file A(using above the same outputstream, 2nd block need to
be written)
4 standby namenode load the above edits and would crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
[
https://issues.apache.org/jira/browse/HDFS-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei reassigned HDFS-16105:
--
Assignee: Hui Fei
> Edit log corruption due to mismatch between fileId and path
> ---
>
> Key: HDFS-16105
> URL: https://issues.apache.org/jira/browse/HDFS-16105
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.2.2, 3.3.1
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
>
> We do stress testing in FUSE with HDFS, Standby Namenode crashes.
> The log is following
> {quote}
> 2021-06-25 17:13:02,953 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
> on operation AddBlockOp [path=/xxx/fiotest_write.354.46,
> penultimateBlock=xxx, lastBlock=xxx, numOfBytes=0}, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File /xxx/fiotest_write.354.46 does not exist.
> {quote}
> The following steps can reproduce it (Illegal writes).
> 1 create file A (fileId X) (1st block is being written)
> 2 rename file A to file B(stile fileId X)
> 3 continue to write file A(using above the same outputstream, 2nd block need
> to be written)
> 4 standby namenode load the above edits and would crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16086) Add volume information to datanode log for tracing
[ https://issues.apache.org/jira/browse/HDFS-16086?focusedWorklogId=617409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617409 ] ASF GitHub Bot logged work on HDFS-16086: - Author: ASF GitHub Bot Created on: 01/Jul/21 05:06 Start Date: 01/Jul/21 05:06 Worklog Time Spent: 10m Work Description: jojochuang commented on pull request #3136: URL: https://github.com/apache/hadoop/pull/3136#issuecomment-871923263 The checkstyle warnings are old, unrelated. Mergeing the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617409) Time Spent: 3h (was: 2h 50m) > Add volume information to datanode log for tracing > -- > > Key: HDFS-16086 > URL: https://issues.apache.org/jira/browse/HDFS-16086 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: CreatingRbw.jpg, Received.jpg > > Time Spent: 3h > Remaining Estimate: 0h > > To keep track of the block in volume, we can add the volume information to > the datanode log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16086) Add volume information to datanode log for tracing
[ https://issues.apache.org/jira/browse/HDFS-16086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16086. Fix Version/s: 3.4.0 Resolution: Fixed > Add volume information to datanode log for tracing > -- > > Key: HDFS-16086 > URL: https://issues.apache.org/jira/browse/HDFS-16086 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: CreatingRbw.jpg, Received.jpg > > Time Spent: 3h 10m > Remaining Estimate: 0h > > To keep track of the block in volume, we can add the volume information to > the datanode log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16086) Add volume information to datanode log for tracing
[ https://issues.apache.org/jira/browse/HDFS-16086?focusedWorklogId=617410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617410 ] ASF GitHub Bot logged work on HDFS-16086: - Author: ASF GitHub Bot Created on: 01/Jul/21 05:07 Start Date: 01/Jul/21 05:07 Worklog Time Spent: 10m Work Description: jojochuang merged pull request #3136: URL: https://github.com/apache/hadoop/pull/3136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617410) Time Spent: 3h 10m (was: 3h) > Add volume information to datanode log for tracing > -- > > Key: HDFS-16086 > URL: https://issues.apache.org/jira/browse/HDFS-16086 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: CreatingRbw.jpg, Received.jpg > > Time Spent: 3h 10m > Remaining Estimate: 0h > > To keep track of the block in volume, we can add the volume information to > the datanode log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
[ https://issues.apache.org/jira/browse/HDFS-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372310#comment-17372310 ] lei w commented on HDFS-16101: -- Thanks [~ayushtkn] for your reply. > Remove unuse variable and IoException in ProvidedStorageMap > --- > > Key: HDFS-16101 > URL: https://issues.apache.org/jira/browse/HDFS-16101 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16101.001.patch > > > Remove unuse variable and IoException in ProvidedStorageMap -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16102) Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to save time
[
https://issues.apache.org/jira/browse/HDFS-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372307#comment-17372307
]
lei w commented on HDFS-16102:
--
Thanks [~hexiaoqiao] for your reply. I will update it.
> Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to
> save time
> --
>
> Key: HDFS-16102
> URL: https://issues.apache.org/jira/browse/HDFS-16102
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16102.001.patch
>
>
> The current logic in removeBlocksAssociatedTo(...) is as follows：
> {code:java}
> void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
> providedStorageMap.removeDatanode(node);
> for (DatanodeStorageInfo storage : node.getStorageInfos()) {
> final Iterator it = storage.getBlockIterator();
> //add the BlockInfos to a new collection as the
> //returned iterator is not modifiable.
> Collection toRemove = new ArrayList<>();
> while (it.hasNext()) {
> toRemove.add(it.next()); // First iteration : to put blocks to
> another collection
> }
> for (BlockInfo b : toRemove) {
> removeStoredBlock(b, node); // Another iteration : to remove blocks
> }
> }
> // ..
> }
> {code}
> In fact , we can use the first iteration to achieve this logic , so should
> we remove the redundant iteration to save time and memory?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16088) Standby NameNode process getLiveDatanodeStorageReport request to reduce Active load
[ https://issues.apache.org/jira/browse/HDFS-16088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372299#comment-17372299 ] tomscut commented on HDFS-16088: Thanks [~hexiaoqiao] for your comment and thoughtful consideration. IMO FSNamesystem#getDatanodeStorageReport is OperationCategory.UNCHECKED ([HDFS-6739|https://issues.apache.org/jira/browse/HDFS-6739]), so we can request redirect to Standby on HA mode. I will add an unit test for this ASAP. > Standby NameNode process getLiveDatanodeStorageReport request to reduce > Active load > --- > > Key: HDFS-16088 > URL: https://issues.apache.org/jira/browse/HDFS-16088 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > As with HDFS-13183, NameNodeConnector#getLiveDatanodeStorageReport() can also > request to SNN to reduce the ANN load. > There are two points that need to be mentioned: > 1. FSNamesystem#getLiveDatanodeStorageReport() is > OperationCategory.UNCHECKED, so we can access SNN directly. > 2. We can share the same UT(testBalancerRequestSBNWithHA) with > NameNodeConnector#getBlocks(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16088) Standby NameNode process getLiveDatanodeStorageReport request to reduce Active load
[ https://issues.apache.org/jira/browse/HDFS-16088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16088: --- Description: As with HDFS-13183, NameNodeConnector#getLiveDatanodeStorageReport() can also request to SNN to reduce the ANN load. There are two points that need to be mentioned: 1. FSNamesystem#getLiveDatanodeStorageReport() is OperationCategory.UNCHECKED, so we can access SNN directly. 2. We can share the same UT(testBalancerRequestSBNWithHA) with NameNodeConnector#getBlocks(). was: As with [HDFS-13183|https://issues.apache.org/jira/browse/HDFS-13183], NameNodeConnector#getLiveDatanodeStorageReport() can also request to SNN to reduce the ANN load. There are two points that need to be mentioned: 1. NameNodeConnector#getLiveDatanodeStorageReport() is OperationCategory.UNCHECKED in FSNamesystem, so we can access SNN directly. 2. We can share the same UT(testBalancerRequestSBNWithHA) with NameNodeConnector#getBlocks(). > Standby NameNode process getLiveDatanodeStorageReport request to reduce > Active load > --- > > Key: HDFS-16088 > URL: https://issues.apache.org/jira/browse/HDFS-16088 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > As with HDFS-13183, NameNodeConnector#getLiveDatanodeStorageReport() can also > request to SNN to reduce the ANN load. > There are two points that need to be mentioned: > 1. FSNamesystem#getLiveDatanodeStorageReport() is > OperationCategory.UNCHECKED, so we can access SNN directly. > 2. We can share the same UT(testBalancerRequestSBNWithHA) with > NameNodeConnector#getBlocks(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[
https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=617355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617355
]
ASF GitHub Bot logged work on HDFS-16104:
-
Author: ASF GitHub Bot
Created on: 01/Jul/21 00:23
Start Date: 01/Jul/21 00:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3160:
URL: https://github.com/apache/hadoop/pull/3160#issuecomment-871813663
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 1m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 47s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 345m 33s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 452m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3160 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 377425d16141 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7a6a6b865e83aa2bee61fc81c1832ceda0a0ecaf |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[jira] [Updated] (HDFS-16090) Fine grained locking for datanodeNetworkCounts
[ https://issues.apache.org/jira/browse/HDFS-16090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HDFS-16090: - Fix Version/s: 3.3.2 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-3.3+ (It didn't go in clean against branch-3.2). Resolving. Thanks for the improvement [~vjasani] . Thanks for reviews [~aajisaka] and [~weichiu] > Fine grained locking for datanodeNetworkCounts > -- > > Key: HDFS-16090 > URL: https://issues.apache.org/jira/browse/HDFS-16090 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2.5h > Remaining Estimate: 0h > > While incrementing DataNode network error count, we lock entire LoadingCache > in order to increment network count of specific host. We should provide fine > grained concurrency for this update because locking entire cache is redundant > and could impact performance while incrementing network count for multiple > hosts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16090) Fine grained locking for datanodeNetworkCounts
[ https://issues.apache.org/jira/browse/HDFS-16090?focusedWorklogId=617246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617246 ] ASF GitHub Bot logged work on HDFS-16090: - Author: ASF GitHub Bot Created on: 30/Jun/21 19:55 Start Date: 30/Jun/21 19:55 Worklog Time Spent: 10m Work Description: saintstack merged pull request #3148: URL: https://github.com/apache/hadoop/pull/3148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617246) Time Spent: 2.5h (was: 2h 20m) > Fine grained locking for datanodeNetworkCounts > -- > > Key: HDFS-16090 > URL: https://issues.apache.org/jira/browse/HDFS-16090 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > While incrementing DataNode network error count, we lock entire LoadingCache > in order to increment network count of specific host. We should provide fine > grained concurrency for this update because locking entire cache is redundant > and could impact performance while incrementing network count for multiple > hosts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16088) Standby NameNode process getLiveDatanodeStorageReport request to reduce Active load
[ https://issues.apache.org/jira/browse/HDFS-16088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372119#comment-17372119 ] Xiaoqiao He commented on HDFS-16088: Thanks [~tomscut] involve me here. It makes sense to me at first glance. However #getLiveDatanodeStorageReport is not quite the same with #getBlocks method IMO, especially it is also used by mover and diskbalancer, I am not sure if it is graceful to request redirect to Standby on HA mode. + It will be more helpful to extent the unit test. Thanks. > Standby NameNode process getLiveDatanodeStorageReport request to reduce > Active load > --- > > Key: HDFS-16088 > URL: https://issues.apache.org/jira/browse/HDFS-16088 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > As with [HDFS-13183|https://issues.apache.org/jira/browse/HDFS-13183], > NameNodeConnector#getLiveDatanodeStorageReport() can also request to SNN to > reduce the ANN load. > There are two points that need to be mentioned: > 1. NameNodeConnector#getLiveDatanodeStorageReport() is > OperationCategory.UNCHECKED in FSNamesystem, so we can access SNN directly. > 2. We can share the same UT(testBalancerRequestSBNWithHA) with > NameNodeConnector#getBlocks(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16093) DataNodes under decommission will still be returned to the client via getLocatedBlocks, so the client may request decommissioning datanodes to read which will cause bad
[ https://issues.apache.org/jira/browse/HDFS-16093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372116#comment-17372116 ] Xiaoqiao He commented on HDFS-16093: -1 to remove DECOMMISSIONING state node from `LocatedBlocks` directly. A. It is possible that all replicas of one block on DECOMMISSIONING node, and client will meet BlockMissing Exception if we do that but actually not as [~sodonnell] mentioned above. B. We would sort DECOMMISSIONING or DECOMMISSIONED to the end of `LocatedBlocks` but not remove directly. > DataNodes under decommission will still be returned to the client via > getLocatedBlocks, so the client may request decommissioning datanodes to read > which will cause badly competation on disk IO. > -- > > Key: HDFS-16093 > URL: https://issues.apache.org/jira/browse/HDFS-16093 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.1 >Reporter: Daniel Ma >Priority: Critical > > DataNodes under decommission will still be returned to the client via > getLocatedBlocks, so the client may request decommissioning datanodes to read > which will cause badly competation on disk IO. > Therefore, datanodes under decommission should be removed from the return > list of getLocatedBlocks api. > !image-2021-06-29-10-50-44-739.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16095) Add lsQuotaList command and getQuotaListing api for hdfs quota
[ https://issues.apache.org/jira/browse/HDFS-16095?focusedWorklogId=617158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617158 ] ASF GitHub Bot logged work on HDFS-16095: - Author: ASF GitHub Bot Created on: 30/Jun/21 16:47 Start Date: 30/Jun/21 16:47 Worklog Time Spent: 10m Work Description: kihwal commented on pull request #3155: URL: https://github.com/apache/hadoop/pull/3155#issuecomment-871567632 It has a potential to hold the fsn/fsd lock for a long time and cause service outage or delays. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617158) Time Spent: 50m (was: 40m) > Add lsQuotaList command and getQuotaListing api for hdfs quota > -- > > Key: HDFS-16095 > URL: https://issues.apache.org/jira/browse/HDFS-16095 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Xiangyi Zhu >Assignee: Xiangyi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Currently hdfs does not support obtaining all quota information. The > administrator may need to check which quotas have been added to a certain > directory, or the quotas of the entire cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16102) Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to save time
[
https://issues.apache.org/jira/browse/HDFS-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372114#comment-17372114
]
Xiaoqiao He commented on HDFS-16102:
Thanks [~lei w] for your report, It seems that your codebase is not the latest
one nor branch trunk. It has been updated for trunk. FYI. Thanks.
> Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to
> save time
> --
>
> Key: HDFS-16102
> URL: https://issues.apache.org/jira/browse/HDFS-16102
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16102.001.patch
>
>
> The current logic in removeBlocksAssociatedTo(...) is as follows：
> {code:java}
> void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
> providedStorageMap.removeDatanode(node);
> for (DatanodeStorageInfo storage : node.getStorageInfos()) {
> final Iterator it = storage.getBlockIterator();
> //add the BlockInfos to a new collection as the
> //returned iterator is not modifiable.
> Collection toRemove = new ArrayList<>();
> while (it.hasNext()) {
> toRemove.add(it.next()); // First iteration : to put blocks to
> another collection
> }
> for (BlockInfo b : toRemove) {
> removeStoredBlock(b, node); // Another iteration : to remove blocks
> }
> }
> // ..
> }
> {code}
> In fact , we can use the first iteration to achieve this logic , so should
> we remove the redundant iteration to save time and memory?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16097) Datanode receives ipc requests will throw NPE when datanode quickly restart
[
https://issues.apache.org/jira/browse/HDFS-16097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372105#comment-17372105
]
Xiaoqiao He commented on HDFS-16097:
Thanks [~lei w] for your report. It is good improvement! Do you mind to offer
what client will meet when request a DataNode with initializing FsDataSets
state. + It will be more helpful if extent unit test. Thanks.
> Datanode receives ipc requests will throw NPE when datanode quickly restart
>
>
> Key: HDFS-16097
> URL: https://issues.apache.org/jira/browse/HDFS-16097
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Environment:
>Reporter: lei w
>Assignee: lei w
>Priority: Major
> Attachments: HDFS-16097.001.patch
>
>
> Datanode receives ipc requests will throw NPE when datanode quickly restart.
> This is because when DN is reStarted, BlockPool is first registered with
> blockPoolManager and then fsdataset is initialized. When BlockPool is
> registered to blockPoolManager without initializing fsdataset, DataNode
> receives an IPC request will throw NPE, because it will call related methods
> provided by fsdataset. The stack exception is as follows：
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:3468)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolServerSideTranslatorPB.initReplicaRecovery(InterDatanodeProtocolServerSideTranslatorPB.java:55)
> at
> org.apache.hadoop.hdfs.protocol.proto.InterDatanodeProtocolProtos$InterDatanodeProtocolService$2.callBlockingMethod(InterDatanodeProtocolProtos.java:3105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:916)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:862)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16097) Datanode receives ipc requests will throw NPE when datanode quickly restart
[
https://issues.apache.org/jira/browse/HDFS-16097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-16097:
---
Status: Patch Available (was: Open)
> Datanode receives ipc requests will throw NPE when datanode quickly restart
>
>
> Key: HDFS-16097
> URL: https://issues.apache.org/jira/browse/HDFS-16097
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Environment:
>Reporter: lei w
>Assignee: lei w
>Priority: Major
> Attachments: HDFS-16097.001.patch
>
>
> Datanode receives ipc requests will throw NPE when datanode quickly restart.
> This is because when DN is reStarted, BlockPool is first registered with
> blockPoolManager and then fsdataset is initialized. When BlockPool is
> registered to blockPoolManager without initializing fsdataset, DataNode
> receives an IPC request will throw NPE, because it will call related methods
> provided by fsdataset. The stack exception is as follows：
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:3468)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolServerSideTranslatorPB.initReplicaRecovery(InterDatanodeProtocolServerSideTranslatorPB.java:55)
> at
> org.apache.hadoop.hdfs.protocol.proto.InterDatanodeProtocolProtos$InterDatanodeProtocolService$2.callBlockingMethod(InterDatanodeProtocolProtos.java:3105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:916)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:862)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16097) Datanode receives ipc requests will throw NPE when datanode quickly restart
[
https://issues.apache.org/jira/browse/HDFS-16097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He reassigned HDFS-16097:
--
Assignee: lei w
> Datanode receives ipc requests will throw NPE when datanode quickly restart
>
>
> Key: HDFS-16097
> URL: https://issues.apache.org/jira/browse/HDFS-16097
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Environment:
>Reporter: lei w
>Assignee: lei w
>Priority: Major
> Attachments: HDFS-16097.001.patch
>
>
> Datanode receives ipc requests will throw NPE when datanode quickly restart.
> This is because when DN is reStarted, BlockPool is first registered with
> blockPoolManager and then fsdataset is initialized. When BlockPool is
> registered to blockPoolManager without initializing fsdataset, DataNode
> receives an IPC request will throw NPE, because it will call related methods
> provided by fsdataset. The stack exception is as follows：
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:3468)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolServerSideTranslatorPB.initReplicaRecovery(InterDatanodeProtocolServerSideTranslatorPB.java:55)
> at
> org.apache.hadoop.hdfs.protocol.proto.InterDatanodeProtocolProtos$InterDatanodeProtocolService$2.callBlockingMethod(InterDatanodeProtocolProtos.java:3105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:916)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:862)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372103#comment-17372103 ] Xiaoqiao He commented on HDFS-16100: Thanks [~g20141821] for your report. Add [~g20141821] to contributors list and assign this issue to him. > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He reassigned HDFS-16100: -- Assignee: wudeyu > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[
https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=617134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617134
]
ASF GitHub Bot logged work on HDFS-16104:
-
Author: ASF GitHub Bot
Created on: 30/Jun/21 16:10
Start Date: 30/Jun/21 16:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3160:
URL: https://github.com/apache/hadoop/pull/3160#issuecomment-871536689
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 23m 11s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 28s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 30s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 26s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 28s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 28s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 29s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | spotbugs | 3m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 1m 33s |
[/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3160/1/artifact/out/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 502 new + 0 unchanged
- 0 fixed = 502 total (was 0) |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 1m 25s |
[jira] [Work logged] (HDFS-16099) Make bpServiceToActive to be volatile
[
https://issues.apache.org/jira/browse/HDFS-16099?focusedWorklogId=617119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617119
]
ASF GitHub Bot logged work on HDFS-16099:
-
Author: ASF GitHub Bot
Created on: 30/Jun/21 15:38
Start Date: 30/Jun/21 15:38
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3159:
URL: https://github.com/apache/hadoop/pull/3159#issuecomment-871510747
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 27s | | trunk passed |
| +1 :green_heart: | compile | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 35s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 433m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3159/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 543m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.blockmanagement.TestErasureCodingCorruption |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372039#comment-17372039 ] lei w commented on HDFS-16083: -- Thanks [~LiJinglun] reply. Take your suggestion and make some changes in v05 . Please review again. > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.patch, > activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w updated HDFS-16083: - Attachment: HDFS-16083.005.patch > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.patch, > activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
[
https://issues.apache.org/jira/browse/HDFS-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372009#comment-17372009
]
Ayush Saxena commented on HDFS-16101:
-
Makes sense to me.
The class is marked private and unstable, so removing the IOException won't be
a problem from compatibility point of view:
{noformat}
@InterfaceAudience.Private
@InterfaceStability.Unstable{noformat}
Have triggered the build, If no complains will commit post that
> Remove unuse variable and IoException in ProvidedStorageMap
> ---
>
> Key: HDFS-16101
> URL: https://issues.apache.org/jira/browse/HDFS-16101
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16101.001.patch
>
>
> Remove unuse variable and IoException in ProvidedStorageMap
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
[ https://issues.apache.org/jira/browse/HDFS-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-16101: Assignee: lei w Status: Patch Available (was: Open) > Remove unuse variable and IoException in ProvidedStorageMap > --- > > Key: HDFS-16101 > URL: https://issues.apache.org/jira/browse/HDFS-16101 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16101.001.patch > > > Remove unuse variable and IoException in ProvidedStorageMap -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
[ https://issues.apache.org/jira/browse/HDFS-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371998#comment-17371998 ] lei w commented on HDFS-16101: -- [~ayushsaxena] Could you give me some advice? > Remove unuse variable and IoException in ProvidedStorageMap > --- > > Key: HDFS-16101 > URL: https://issues.apache.org/jira/browse/HDFS-16101 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: lei w >Priority: Minor > Attachments: HDFS-16101.001.patch > > > Remove unuse variable and IoException in ProvidedStorageMap -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16095) Add lsQuotaList command and getQuotaListing api for hdfs quota
[
https://issues.apache.org/jira/browse/HDFS-16095?focusedWorklogId=617027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617027
]
ASF GitHub Bot logged work on HDFS-16095:
-
Author: ASF GitHub Bot
Created on: 30/Jun/21 12:54
Start Date: 30/Jun/21 12:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3155:
URL: https://github.com/apache/hadoop/pull/3155#issuecomment-871378621
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 32s | | trunk passed |
| +1 :green_heart: | compile | 22m 35s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 39s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 9m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 32s | | the patch passed |
| +1 :green_heart: | compile | 21m 59s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | cc | 21m 59s |
[/results-compile-cc-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/2/artifact/out/results-compile-cc-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 34 new + 289 unchanged - 34
fixed = 323 total (was 323) |
| -1 :x: | javac | 21m 59s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/2/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1981 unchanged - 0
fixed = 1982 total (was 1981) |
| +1 :green_heart: | compile | 19m 12s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | cc | 19m 12s |
[/results-compile-cc-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/2/artifact/out/results-compile-cc-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 7 new + 316
unchanged - 7 fixed = 323 total (was 323) |
| -1 :x: | javac | 19m 12s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/2/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 3 new + 1855
unchanged - 2 fixed = 1858 total (was 1857) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/2/artifact/out/blanks-eol.txt)
| The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 3s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/2/artifact/out/results-checkstyle-root.txt)
|
[jira] [Work logged] (HDFS-16086) Add volume information to datanode log for tracing
[ https://issues.apache.org/jira/browse/HDFS-16086?focusedWorklogId=616994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616994 ] ASF GitHub Bot logged work on HDFS-16086: - Author: ASF GitHub Bot Created on: 30/Jun/21 10:50 Start Date: 30/Jun/21 10:50 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3136: URL: https://github.com/apache/hadoop/pull/3136#issuecomment-871296935 Hi @ayushtkn , could you please take a quick look at this. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616994) Time Spent: 2h 50m (was: 2h 40m) > Add volume information to datanode log for tracing > -- > > Key: HDFS-16086 > URL: https://issues.apache.org/jira/browse/HDFS-16086 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: CreatingRbw.jpg, Received.jpg > > Time Spent: 2h 50m > Remaining Estimate: 0h > > To keep track of the block in volume, we can add the volume information to > the datanode log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=616992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616992 ] ASF GitHub Bot logged work on HDFS-16104: - Author: ASF GitHub Bot Created on: 30/Jun/21 10:49 Start Date: 30/Jun/21 10:49 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3160: URL: https://github.com/apache/hadoop/pull/3160#issuecomment-871296156 > LGTM Thanks @ayushtkn for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616992) Time Spent: 20m (was: 10m) > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16104: -- Labels: pull-request-available (was: ) > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=616990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616990 ] ASF GitHub Bot logged work on HDFS-16104: - Author: ASF GitHub Bot Created on: 30/Jun/21 10:27 Start Date: 30/Jun/21 10:27 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #3160: URL: https://github.com/apache/hadoop/pull/3160 JIRA: [HDFS-16104](https://issues.apache.org/jira/browse/HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616990) Remaining Estimate: 0h Time Spent: 10m > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
tomscut created HDFS-16104: -- Summary: Remove unused parameter and fix java doc for DiskBalancerCLI Key: HDFS-16104 URL: https://issues.apache.org/jira/browse/HDFS-16104 Project: Hadoop HDFS Issue Type: Wish Reporter: tomscut Assignee: tomscut Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16103) mvn test failed about hadoop@3.2.1
[
https://issues.apache.org/jira/browse/HDFS-16103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shixijun updated HDFS-16103:
Affects Version/s: 3.2.1
Description:
{panel:title=mvn test failed about hadoop@3.2.1}
mvn test failed
{panel}
[root@localhost spack-src]# mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home:
/home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/maven-3.6.3-fpgpwvz7es5yiaz2tez2pnlilrcatuvg
Java version: 1.8.0_191, vendor: AdoptOpenJdk, runtime:
/home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/openjdk-1.8.0_191-b12-fidptihybskgklbjoo4lagkacm6n6lod/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "4.18.0-80.el8.aarch64", arch: "aarch64", family:
"unix"
[root@localhost spack-src]# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.191-b12, mixed mode)
[root@localhost spack-src]# mvn test
……
[INFO] Running org.apache.hadoop.tools.TestCommandShell
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111 s
- in org.apache.hadoop.tools.TestCommandShell
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
Should throw IOException
[ERROR]
TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288
Should throw IOException
[ERROR]
TestFileUtil.testFailFullyDelete:446->validateAndSetWritablePermissions:422 The
directory xSubDir *should* not have been deleted. expected: but
was:
[ERROR]
TestFileUtil.testFailFullyDeleteContents:525->validateAndSetWritablePermissions:422
The directory xSubDir *should* not have been deleted. expected: but
was:
[ERROR] TestFileUtil.testGetDU:571
[ERROR] TestFsShellCopy.testPutSrcDirNoPerm:627->shellRun:80 expected:<1> but
was:<0>
[ERROR] TestFsShellCopy.testPutSrcFileNoPerm:652->shellRun:80 expected:<1>
but was:<0>
[ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking
for build/test/temp/RELATIVE1 in
build/test/temp/RELATIVE0/block995011826146306285.tmp - FAILED!
[ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking
for
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block792666236482175348.tmp
- FAILED!
[ERROR] TestLocalDirAllocator.test0:141->validateTempDirCreation:109 Checking
for
file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5124616846677903649.tmp
- FAILED!
[ERROR]
TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109
Checking for build/test/temp/RELATIVE2 in
build/test/temp/RELATIVE1/block1176062344115776027.tmp - FAILED!
[ERROR]
TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
Checking for
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block3514694215643608527.tmp
- FAILED!
[ERROR]
TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
Checking for
file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block883026101475466701.tmp
- FAILED!
[ERROR]
TestLocalDirAllocator.testRWBufferDirBecomesRO:219->validateTempDirCreation:109
Checking for build/test/temp/RELATIVE3 in
build/test/temp/RELATIVE4/block2198073115547564040.tmp - FAILED!
[ERROR]
TestLocalDirAllocator.testRWBufferDirBecomesRO:219->validateTempDirCreation:109
Checking for
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE3
in
[jira] [Created] (HDFS-16103) mvn test failed about hadoop@3.2.1
shixijun created HDFS-16103:
---
Summary: mvn test failed about hadoop@3.2.1
Key: HDFS-16103
URL: https://issues.apache.org/jira/browse/HDFS-16103
Project: Hadoop HDFS
Issue Type: Bug
Reporter: shixijun
{panel:title=mvn test failed about hadoop@3.2.1}
mvn test failed
{panel}
[root@localhost spack-src]# mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home:
/home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/maven-3.6.3-fpgpwvz7es5yiaz2tez2pnlilrcatuvg
Java version: 1.8.0_191, vendor: AdoptOpenJdk, runtime:
/home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/openjdk-1.8.0_191-b12-fidptihybskgklbjoo4lagkacm6n6lod/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "4.18.0-80.el8.aarch64", arch: "aarch64", family:
"unix"
[root@localhost spack-src]# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.191-b12, mixed mode)
[root@localhost spack-src]# mvn test
……
[INFO] Running org.apache.hadoop.tools.TestCommandShell
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111 s
- in org.apache.hadoop.tools.TestCommandShell
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
Should throw IOException
[ERROR]
TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288
Should throw IOException
[ERROR]
TestFileUtil.testFailFullyDelete:446->validateAndSetWritablePermissions:422 The
directory xSubDir *should* not have been deleted. expected: but
was:
[ERROR]
TestFileUtil.testFailFullyDeleteContents:525->validateAndSetWritablePermissions:422
The directory xSubDir *should* not have been deleted. expected: but
was:
[ERROR] TestFileUtil.testGetDU:571
[ERROR] TestFsShellCopy.testPutSrcDirNoPerm:627->shellRun:80 expected:<1> but
was:<0>
[ERROR] TestFsShellCopy.testPutSrcFileNoPerm:652->shellRun:80 expected:<1>
but was:<0>
[ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking
for build/test/temp/RELATIVE1 in
build/test/temp/RELATIVE0/block995011826146306285.tmp - FAILED!
[ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking
for
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block792666236482175348.tmp
- FAILED!
[ERROR] TestLocalDirAllocator.test0:141->validateTempDirCreation:109 Checking
for
file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5124616846677903649.tmp
- FAILED!
[ERROR]
TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109
Checking for build/test/temp/RELATIVE2 in
build/test/temp/RELATIVE1/block1176062344115776027.tmp - FAILED!
[ERROR]
TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
Checking for
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block3514694215643608527.tmp
- FAILED!
[ERROR]
TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
Checking for
file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2
in
/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block883026101475466701.tmp
- FAILED!
[ERROR]
TestLocalDirAllocator.testRWBufferDirBecomesRO:219->validateTempDirCreation:109
Checking for build/test/temp/RELATIVE3 in
build/test/temp/RELATIVE4/block2198073115547564040.tmp - FAILED!
[ERROR]
TestLocalDirAllocator.testRWBufferDirBecomesRO:219->validateTempDirCreation:109
Checking for
[jira] [Commented] (HDFS-16098) ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDFS-16098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371921#comment-17371921 ] Stephen O'Donnell commented on HDFS-16098: -- HDFS-13728 and HDFS-13727 may be relevant. > ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException > --- > > Key: HDFS-16098 > URL: https://issues.apache.org/jira/browse/HDFS-16098 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Affects Versions: 2.6.0 > Environment: VERSION info: > Hadoop 2.6.0-cdh5.14.4 >Reporter: wangyanfu >Priority: Blocker > Labels: diskbalancer > Fix For: 2.6.0 > > Attachments: on-branch-3.1.jpg > > Original Estimate: 504h > Remaining Estimate: 504h > > when i tried to run > hdfs diskbalancer -plan $(hostname -f) > > > > i get this notice: > 21/06/30 11:30:41 ERROR tools.DiskBalancerCLI: > java.lang.IllegalArgumentException > > then i tried write the real hostname into my command , not work and same > error notice > i also tried use --plan instead of -plan , not work and same error notice > i found this > [link|https://community.cloudera.com/t5/Support-Questions/Error-trying-to-balance-disks-on-node/m-p/59989#M54850] > but there's no resolve solution , can somebody help me? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-16083: --- Attachment: HDFS-16083.004.patch Status: Patch Available (was: Open) Re-submit v04. > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[
https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371915#comment-17371915
]
Jinglun commented on HDFS-16083:
Hi [~lei w], thanks your patch, some comments.
In EditLogTailer.java:
# I prefer using `shouldRollLog` instead of avoidTriggerActiveLogRoll.
{code:java}
if (shouldRollLog && tooLongSinceLastLoad() &&
lastRollTriggerTxId < lastLoadedTxnId) {{code}
In TestStandbyRollEditsLogOnly.java:
# The test case and setup method should not be static.
# We need a License for the new file.
In TestStandbyRollEditsLogOnly#testOnlyStandbyRollEditlog:
# When you compare observerRollTimeMs1, could you use assertEquals instead of
assertTrue.
# The message of the assert should be more specific. Something like: "Standby
should roll the log." and "The observer is not expected to roll the log."
# I'd prefer using standbyInitialRollTime and standbyLastRollTime instead of
using numbers standbyRollTimeMs1 and standbyRollTimeMs2.
# The sleep time is too long, can we make it faster ?
In TestStandbyRollEditsLogOnly#testTransObToStandbyThenRollLog:
# It fails, could you give it a check.
# The verify logic is very like testOnlyStandbyRollEditlog, can we extract the
same part as a new method.
# The idea of this test is good. We can transition the state and verify roll
edit more times. May be do it 3 times ?
There is also some checkstyle issue. Please follow jenkins suggestions. I'll
re-submit v03 as v04 to trigger jenkins.
> Forbid Observer NameNode trigger active namenode log roll
> --
>
> Key: HDFS-16083
> URL: https://issues.apache.org/jira/browse/HDFS-16083
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch,
> HDFS-16083.003.patch, activeRollEdits.png
>
>
> When the Observer NameNode is turned on in the cluster, the Active NameNode
> will receive rollEditLog RPC requests from the Standby NameNode and Observer
> NameNode in a short time. Observer NameNode's rollEditLog request is a
> repetitive operation, so should we forbid Observer NameNode trigger active
> namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5
> minutes) and active NameNode receives rollEditLog RPC as shown in
> activeRollEdits.png
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16102) Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to save time
[
https://issues.apache.org/jira/browse/HDFS-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lei w updated HDFS-16102:
-
Attachment: HDFS-16102.001.patch
> Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to
> save time
> --
>
> Key: HDFS-16102
> URL: https://issues.apache.org/jira/browse/HDFS-16102
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16102.001.patch
>
>
> The current logic in removeBlocksAssociatedTo(...) is as follows：
> {code:java}
> void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
> providedStorageMap.removeDatanode(node);
> for (DatanodeStorageInfo storage : node.getStorageInfos()) {
> final Iterator it = storage.getBlockIterator();
> //add the BlockInfos to a new collection as the
> //returned iterator is not modifiable.
> Collection toRemove = new ArrayList<>();
> while (it.hasNext()) {
> toRemove.add(it.next()); // First iteration : to put blocks to
> another collection
> }
> for (BlockInfo b : toRemove) {
> removeStoredBlock(b, node); // Another iteration : to remove blocks
> }
> }
> // ..
> }
> {code}
> In fact , we can use the first iteration to achieve this logic , so should
> we remove the redundant iteration to save time and memory?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16102) Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to save time
[
https://issues.apache.org/jira/browse/HDFS-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lei w updated HDFS-16102:
-
Description:
The current logic in removeBlocksAssociatedTo(...) is as follows：
{code:java}
void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
providedStorageMap.removeDatanode(node);
for (DatanodeStorageInfo storage : node.getStorageInfos()) {
final Iterator it = storage.getBlockIterator();
//add the BlockInfos to a new collection as the
//returned iterator is not modifiable.
Collection toRemove = new ArrayList<>();
while (it.hasNext()) {
toRemove.add(it.next()); // First iteration : to put blocks to another
collection
}
for (BlockInfo b : toRemove) {
removeStoredBlock(b, node); // Another iteration : to remove blocks
}
}
// ..
}
{code}
In fact , we can use the first iteration to achieve this logic , so should we
remove the redundant iteration to save time and memory?
was:
The current logic in removeBlocksAssociatedTo(...) is as follows：
{code:java}
void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
providedStorageMap.removeDatanode(node);
for (DatanodeStorageInfo storage : node.getStorageInfos()) {
final Iterator it = storage.getBlockIterator();
//add the BlockInfos to a new collection as the
//returned iterator is not modifiable.
Collection toRemove = new ArrayList<>();
while (it.hasNext()) {
toRemove.add(it.next()); // First iteration : to put blocks to another
collection
}
for (BlockInfo b : toRemove) {
removeStoredBlock(b, node); // Another iteration : to remove blocks
}
}
// ..
}
{code}
In fact , we can use the first iteration to achieve this logic , so should we
remove the redundant iteration to save time?
> Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to
> save time
> --
>
> Key: HDFS-16102
> URL: https://issues.apache.org/jira/browse/HDFS-16102
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16102.001.patch
>
>
> The current logic in removeBlocksAssociatedTo(...) is as follows：
> {code:java}
> void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
> providedStorageMap.removeDatanode(node);
> for (DatanodeStorageInfo storage : node.getStorageInfos()) {
> final Iterator it = storage.getBlockIterator();
> //add the BlockInfos to a new collection as the
> //returned iterator is not modifiable.
> Collection toRemove = new ArrayList<>();
> while (it.hasNext()) {
> toRemove.add(it.next()); // First iteration : to put blocks to
> another collection
> }
> for (BlockInfo b : toRemove) {
> removeStoredBlock(b, node); // Another iteration : to remove blocks
> }
> }
> // ..
> }
> {code}
> In fact , we can use the first iteration to achieve this logic , so should
> we remove the redundant iteration to save time and memory?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16102) Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to save time
lei w created HDFS-16102:
Summary: Remove redundant iteration in
BlockManager#removeBlocksAssociatedTo(...) to save time
Key: HDFS-16102
URL: https://issues.apache.org/jira/browse/HDFS-16102
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: lei w
Assignee: lei w
The current logic in removeBlocksAssociatedTo(...) is as follows：
{code:java}
void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
providedStorageMap.removeDatanode(node);
for (DatanodeStorageInfo storage : node.getStorageInfos()) {
final Iterator it = storage.getBlockIterator();
//add the BlockInfos to a new collection as the
//returned iterator is not modifiable.
Collection toRemove = new ArrayList<>();
while (it.hasNext()) {
toRemove.add(it.next()); // First iteration : to put blocks to another
collection
}
for (BlockInfo b : toRemove) {
removeStoredBlock(b, node); // Another iteration : to remove blocks
}
}
// ..
}
{code}
In fact , we can use the first iteration to achieve this logic , so should we
remove the redundant iteration to save time?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wudeyu updated HDFS-16100: -- Attachment: HDFS-16100.patch Affects Version/s: 3.3.1 Status: Patch Available (was: Open) > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wudeyu updated HDFS-16100: -- Issue Type: Improvement (was: Wish) > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
[ https://issues.apache.org/jira/browse/HDFS-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w updated HDFS-16101: - Attachment: HDFS-16101.001.patch > Remove unuse variable and IoException in ProvidedStorageMap > --- > > Key: HDFS-16101 > URL: https://issues.apache.org/jira/browse/HDFS-16101 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: lei w >Priority: Minor > Attachments: HDFS-16101.001.patch > > > Remove unuse variable and IoException in ProvidedStorageMap -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
lei w created HDFS-16101: Summary: Remove unuse variable and IoException in ProvidedStorageMap Key: HDFS-16101 URL: https://issues.apache.org/jira/browse/HDFS-16101 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: lei w Remove unuse variable and IoException in ProvidedStorageMap -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16090) Fine grained locking for datanodeNetworkCounts
[ https://issues.apache.org/jira/browse/HDFS-16090?focusedWorklogId=616950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616950 ] ASF GitHub Bot logged work on HDFS-16090: - Author: ASF GitHub Bot Created on: 30/Jun/21 07:40 Start Date: 30/Jun/21 07:40 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3148: URL: https://github.com/apache/hadoop/pull/3148#issuecomment-871170905 Thanks for your reviews @jojochuang @aajisaka. Could you please help merge this PR? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616950) Time Spent: 2h 20m (was: 2h 10m) > Fine grained locking for datanodeNetworkCounts > -- > > Key: HDFS-16090 > URL: https://issues.apache.org/jira/browse/HDFS-16090 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > While incrementing DataNode network error count, we lock entire LoadingCache > in order to increment network count of specific host. We should provide fine > grained concurrency for this update because locking entire cache is redundant > and could impact performance while incrementing network count for multiple > hosts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16086) Add volume information to datanode log for tracing
[ https://issues.apache.org/jira/browse/HDFS-16086?focusedWorklogId=616947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616947 ] ASF GitHub Bot logged work on HDFS-16086: - Author: ASF GitHub Bot Created on: 30/Jun/21 07:16 Start Date: 30/Jun/21 07:16 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3136: URL: https://github.com/apache/hadoop/pull/3136#issuecomment-871157070 Hi @jojochuang , could you please take a look again? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616947) Time Spent: 2h 40m (was: 2.5h) > Add volume information to datanode log for tracing > -- > > Key: HDFS-16086 > URL: https://issues.apache.org/jira/browse/HDFS-16086 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: CreatingRbw.jpg, Received.jpg > > Time Spent: 2h 40m > Remaining Estimate: 0h > > To keep track of the block in volume, we can add the volume information to > the datanode log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16096) Delete useless method DirectoryWithQuotaFeature#setQuota
[ https://issues.apache.org/jira/browse/HDFS-16096?focusedWorklogId=616945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616945 ] ASF GitHub Bot logged work on HDFS-16096: - Author: ASF GitHub Bot Created on: 30/Jun/21 07:12 Start Date: 30/Jun/21 07:12 Worklog Time Spent: 10m Work Description: jojochuang merged pull request #3156: URL: https://github.com/apache/hadoop/pull/3156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616945) Time Spent: 1h 20m (was: 1h 10m) > Delete useless method DirectoryWithQuotaFeature#setQuota > > > Key: HDFS-16096 > URL: https://issues.apache.org/jira/browse/HDFS-16096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xiangyi Zhu >Assignee: Xiangyi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Delete useless method DirectoryWithQuotaFeature#setQuota. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16096) Delete useless method DirectoryWithQuotaFeature#setQuota
[ https://issues.apache.org/jira/browse/HDFS-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16096. Resolution: Fixed Merged the PR. Thanks [~zhuxiangyi] and [~vjasani] for the review. > Delete useless method DirectoryWithQuotaFeature#setQuota > > > Key: HDFS-16096 > URL: https://issues.apache.org/jira/browse/HDFS-16096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xiangyi Zhu >Assignee: Xiangyi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Delete useless method DirectoryWithQuotaFeature#setQuota. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuyan Zhang updated HDFS-16099: Description: BPOfferService#bpServiceToActive is not volatile, which may cause _CommandProcessingThread_ to get the out-of-date active namenode. When a failover occurs, the old ANN's _CommandProcessingThread_ may read the outdated BPOfferService#bpServiceToActive and execute the NN's command. At this time, if the new ANN's _CommandProcessingThread_ reads the new value of bpServiceToActive, split brain will occur; otherwise, the new ANN's commands cannot be executed normally, which is also unacceptable. was:BPOfferService#bpServiceToActive is not volatile, which may cause _commandProcessingThread_ to get an out-of-date active namenode. > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _CommandProcessingThread_ to get the out-of-date active namenode. > When a failover occurs, the old ANN's _CommandProcessingThread_ may read the > outdated BPOfferService#bpServiceToActive and execute the NN's command. At > this time, if the new ANN's _CommandProcessingThread_ reads the new value of > bpServiceToActive, split brain will occur; otherwise, the new ANN's commands > cannot be executed normally, which is also unacceptable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16100) HA: Improve performance of Standby node transition to Active
wudeyu created HDFS-16100: - Summary: HA: Improve performance of Standby node transition to Active Key: HDFS-16100 URL: https://issues.apache.org/jira/browse/HDFS-16100 Project: Hadoop HDFS Issue Type: Wish Components: namenode Reporter: wudeyu pendingDNMessages in Standby is used to support process postponed block reports. Block reports in pendingDNMessages would be processed: # If GS of replica is in the future, Standby Node will process it when corresponding edit log(e.g add_block) is loaded. # If replica is corrupted, Standby Node will process it while it transfer to Active. # If DataNode is removed, corresponding of block reports will be removed in pendingDNMessages. Obviously, if num of corrupted replica grows, more time cost during transferring. In out situation, there're 60 millions block reports in pendingDNMessages before transfer. Processing block reports cost almost 7mins and it's killed by zkfc. The replica state of the most block reports is RBW with wrong GS(less than storedblock in Standby Node). In my opinion, Standby Node could ignore the block reports that replica state is RBW with wrong GS. Because Active node/DataNode will remove it later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16099: -- Labels: pull-request-available (was: ) > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _commandProcessingThread_ to get an out-of-date active namenode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16099 started by Shuyan Zhang. --- > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _commandProcessingThread_ to get an out-of-date active namenode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?focusedWorklogId=616932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-616932 ] ASF GitHub Bot logged work on HDFS-16099: - Author: ASF GitHub Bot Created on: 30/Jun/21 06:32 Start Date: 30/Jun/21 06:32 Worklog Time Spent: 10m Work Description: zhangshuyan0 opened a new pull request #3159: URL: https://github.com/apache/hadoop/pull/3159 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 616932) Remaining Estimate: 0h Time Spent: 10m > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _commandProcessingThread_ to get an out-of-date active namenode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16099) Make bpServiceToActive to be volatile
Shuyan Zhang created HDFS-16099: --- Summary: Make bpServiceToActive to be volatile Key: HDFS-16099 URL: https://issues.apache.org/jira/browse/HDFS-16099 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Shuyan Zhang Assignee: Shuyan Zhang BPOfferService#bpServiceToActive is not volatile, which may cause _commandProcessingThread_ to get an out-of-date active namenode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
