[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041216#comment-15041216
]
Xiao Chen commented on HDFS-9313:
-
Thanks again Zhe for the commit!
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0, 2.7.3, 2.6.4
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.branch26.patch,
> HDFS-9313.branch27.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041203#comment-15041203
]
Zhe Zhang commented on HDFS-9313:
-
Thanks Xiao! +1 on the patches. I just committed to branch-2.7 and branch-2.6.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0, 2.7.3, 2.6.4
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.branch26.patch,
> HDFS-9313.branch27.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038479#comment-15038479
]
Xiao Chen commented on HDFS-9313:
-
As [discussed in
HDFS-8647|https://issues.apache.org/jira/browse/HDFS-8647?focusedCommentId=15032747=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15032747],
I'm providing branch2.6/2.7 patch for this JIRA.
Cherry-picking from branch2, the backport was clean. But I had to fix a compile
error about {{HdfsServerConstants}} due to file change. Additionally I removed
the logging change, in order for the code to run. (Otherwise error at run time:
(java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerAdapter cannot be cast
to org.apache.commons.logging.impl.Log4JLogger)).
Locally tested both patches compile and passes TestReplicationPolicy.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.branch26.patch,
> HDFS-9313.branch27.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987027#comment-14987027
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #566 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/566/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987075#comment-14987075
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985638#comment-14985638
]
Zhe Zhang commented on HDFS-9313:
-
Thanks Ming for explaining this. I agree {{break}} is the right logic here. +1
on the latest patch.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986857#comment-14986857
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #631 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/631/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986819#comment-14986819
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1354 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1354/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986793#comment-14986793
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2561 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2561/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986693#comment-14986693
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8746 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8746/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986637#comment-14986637
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #620 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/620/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975815#comment-14975815
]
Hadoop QA commented on HDFS-9313:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 25m 4s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 10m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 14m 7s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 31s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 3s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 2m 5s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 44s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 37s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 67m 32s | Tests failed in hadoop-hdfs. |
| | | 130m 8s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.TestStorageRestore |
| | hadoop.hdfs.server.namenode.TestParallelImageWrite |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.server.namenode.TestProcessCorruptBlocks |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.security.token.block.TestBlockToken |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestAddBlockRetry |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.namenode.TestNameNodeResourceChecker |
| | hadoop.hdfs.server.namenode.TestXAttrConfigFlag |
| | hadoop.hdfs.security.TestDelegationToken |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768868/HDFS-9313.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 96677be |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13212/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13212/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13212/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13212/console |
This message was automatically generated.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976685#comment-14976685
]
Hadoop QA commented on HDFS-9313:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 21m 6s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 9m 11s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 3s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 40s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 48s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 55s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 42s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 55m 9s | Tests failed in hadoop-hdfs. |
| | | 108m 46s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768993/HDFS-9313-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / bcb2386 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13225/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13225/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13225/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13225/console |
This message was automatically generated.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976853#comment-14976853
]
Ming Ma commented on HDFS-9313:
---
Thanks [~zhz]. If it continues, given no state has changed or no alternative
approach is taken, it will just keep looping.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976794#comment-14976794
]
Zhe Zhang commented on HDFS-9313:
-
Thanks Ming for the fix. I haven't dived into full details of the two
{{chooseReplicaToDelete}} methods, but a quick question is why we should
{{break}} instead of {{continue}} when seeing a null value?
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975453#comment-14975453
]
Mingliang Liu commented on HDFS-9313:
-
Thanks for filing and working on this, [~mingma]. I think the patch makes sense
to me. The warning is much better than a NPE.
{code}
+// no replica can't be chosen as the excessive replica as
{code}
Do you mean "no replica *can* be chosen as the excessive replica as"?
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975592#comment-14975592
]
Walter Su commented on HDFS-9313:
-
I'm ok that adding a {{null}} check. However, I don't think it's enough to
address the scenario here, In the test case, you added 1 SSD + 3 DISKs. As you
said in the patch,
{code}
1040// In this case,
1041// no replica can't be chosen as the excessive replica as
1042// chooseReplicasToDelete only considers storages[4] and
storages[5] that
1043// are the same rack. But neither's storage type is SSD.
{code}
If we choose nothing, the replica on SSD won't be deleted. And I remember,
{{Mover}} won't do it neither, since the existings contains the expected.
Instead of choosing nothing, we should choose the SSD, since the remaining 3
DISKs are already on enough racks.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975626#comment-14975626
]
Brahma Reddy Battula commented on HDFS-9313:
Thanks [~mingma] for reporting this.. I too agree.. +1 (non-binding)
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975595#comment-14975595
]
Mingliang Liu commented on HDFS-9313:
-
Will this be covered by [HDFS-9314]?
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975603#comment-14975603
]
Walter Su commented on HDFS-9313:
-
yes.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975612#comment-14975612
]
Ming Ma commented on HDFS-9313:
---
The question is, even after HDFS-9314 is fixed we might still run into other
scenarios where no excess replica can't be picked. It becomes more likely we
continue to add new block placement policies; thus adding null check is more of
defense in depth.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975620#comment-14975620
]
Mingliang Liu commented on HDFS-9313:
-
Agreed.
+1 (non-binding) pending on Jenkins.
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
