[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938708#comment-14938708
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938516#comment-14938516
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938139#comment-14938139
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937919#comment-14937919
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937666#comment-14937666
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937416#comment-14937416
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8548/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937220#comment-14937220
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628173#comment-14628173
]
Yi Liu commented on HDFS-8619:
--
I found we should also fix {{BlockManager#InvalidateBlocks}} for striped block,
filed HDFS-8781.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628134#comment-14628134
]
Yi Liu commented on HDFS-8619:
--
Looks good to me too, thanks Jing!
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627656#comment-14627656
]
Walter Su commented on HDFS-8619:
-
LGTM. +1
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598322#comment-14598322
]
Zhe Zhang commented on HDFS-8619:
-
Thanks Jing for splitting the patch for HDFS-8652!
bq. Please let me know if you need any help on either development or code
review.
Thanks for offering the help! It would be great if you can help review
HDFDS-8623 and HDFS-8653. HDFS-8623 might have some overlap with HDFS-8652
patch. Depending on how they go, I'm happy to rebase either patch.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Auto-Re: [jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
您的邮件已收到！谢谢！
Auto-Re: [jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
您的邮件已收到！谢谢！
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598086#comment-14598086
]
Jing Zhao commented on HDFS-8619:
-
Thanks for the review, Nicholas and Walter! I've created HDFS-8652 to make
changes in trunk. I will update the patch to include changes suggested by
[~walter.k.su]. Since merging the changes from trunk to the feature branch may
take some time, we can continue using this jira to fix remaining issues on the
blockmanager side for handling striped blocks.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597597#comment-14597597
]
Walter Su commented on HDFS-8619:
-
2. {{SafeReplication}} is correct in {{incrementSafeBlockCount}}, but is
incorrect in {{decrementSafeBlockCount}}
3.
{code}
//FSNameSystem#listCorruptFileBlocks(..)
if (inode != null && blockManager.countNodes(blk).liveReplicas() == 0) {
{code}
We should compare liveReplicas with expected data block number(considering file
smaller than a full stripe). Please take a look at HDFS-8567. It add a helper
function in BlockInfoStriped to get expected data block number.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597566#comment-14597566
]
Walter Su commented on HDFS-8619:
-
bq. why removing the if-condition below? Is the condition always true?
I think it is.
The change to {{markBlockAsCorrupt}} and {{invalidateBlock}} is valid for
striped block. One issue:
1. fsck identifies corrupted file by {{LocatedBlock#corrupt}}.
{{LocatedBlock#corrupt}} is from
{code}
//BlockManager#createLocatedBlock(final BlockInfo blk, ...
final boolean isCorrupt = numCorruptNodes != 0 && numCorruptNodes == numNodes;
{code}
Assume blockGroup size is 9, and 8 of them is corrupted, so it's impossible to
recover. In this case, {{isCorrupt}} should be true so fsck can identifies this
file.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597211#comment-14597211
]
Jing Zhao commented on HDFS-8619:
-
bq. Hopefully also to split it to functional pieces like "Support EC zones",
"Allocate and persist striped blocks in NameNode", ""Add striped block support
in INodeFile.".
Yeah, this makes sense to me. Please let me know if you need any help on either
development or code review.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597142#comment-14597142
]
Zhe Zhang commented on HDFS-8619:
-
bq. I guess we still need this jira for adding the striped block logic and the
tests.
Sure. I see at least the tests are specific to striped blocks.
bq. Besides, we now have merged quite a few changes to trunk, any plan for
merging trunk changes to the HDFS-7285 feature branch?
Thanks for bringing up the question Jing. These days I'm mostly focused on this
front. Since the merged changes are quite big, I'm rebasing the entire
consolidated HDFS-7285 patch instead of individual patches. Late last week I
finished a first round of rebasing which is quite rough. I posted a [comment
https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14593827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14593827]
but I guess it is hidden in between Hudson messages (could you help delete
them?). I'm working on a second round of rebase to include all changes.
Hopefully also to split it to functional pieces like "Support EC zones",
"Allocate and persist striped blocks in NameNode", ""Add striped block support
in INodeFile.". I'll post a new rebased patch soon.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597131#comment-14597131
]
Tsz Wo Nicholas Sze commented on HDFS-8619:
---
Patch looks good in general. I agree that we should do most of the changes in
trunk.
- Just a question, why removing the if-condition below? Is the condition
always true?
{code}
//BlockManager.invalidateBlock(..)
-} else if (nr.liveReplicas() >= 1) {
+} else {
{code}
- Let's move numCorruptReplicas from BlockManager to BlockManagerTestUtil.
- See also if we could move getCorruptReplicaBlockIds from CorruptReplicasMap
to BlockManagerTestUtil or some other class in test.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596967#comment-14596967
]
Jing Zhao commented on HDFS-8619:
-
Besides, we now have merged quite a few changes to trunk, any plan for merging
trunk changes to the HDFS-7285 feature branch?
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596959#comment-14596959
]
Jing Zhao commented on HDFS-8619:
-
No. I guess we still need this jira for adding the striped block logic and the
tests.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596933#comment-14596933
]
Zhe Zhang commented on HDFS-8619:
-
Thanks Jing! I meant to say all changes, including the main
{{CorruptReplicasMap}} change LGTM overall and look applicable for trunk.
Should we just retarget the JIRA to trunk?
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596932#comment-14596932
]
Zhe Zhang commented on HDFS-8619:
-
Thanks Jing! I meant to say all changes, including the main
{{CorruptReplicasMap}} change LGTM overall and look applicable for trunk.
Should we just retarget the JIRA to trunk?
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596905#comment-14596905
]
Jing Zhao commented on HDFS-8619:
-
Thanks for the review, Zhe! Sure, I will separate some refactoring out for
trunk.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596846#comment-14596846
]
Zhe Zhang commented on HDFS-8619:
-
A quick comment is maybe we should consider targeting this for trunk? I haven't
finished reviewing the entire patch, and I see the following changes besides
the main change mentioned above:
# A new {{hasNoDataNodes}} logic.
# A {{Block->BlockInfo}} refactor for {{postponedMisreplicatedBlocks}}.
# Refactor of {{invalidateBlock}} to take counted nodes as input instead of
counting again.
# General code cleanups
All changes LGTM overall, and all look applicable against trunk (except for the
tests).
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596799#comment-14596799
]
Zhe Zhang commented on HDFS-8619:
-
Thanks Jing for the work! I think the analysis makes sense. I will review the
patch later today.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594201#comment-14594201
]
Jing Zhao commented on HDFS-8619:
-
The scenario mentioned in the description has been fixed by HDFS-8543: the
{{countNodes}} method actually checks {{excessReplicateMap}} for the excess
replica information, which is correctly updated with the fix from HDFS-8543.
However, the corrupted internal blocks currently cannot be correctly tracked or
counted. Specifically, the {{CorruptReplicasMap}} should track all the DNs with
corrupted internal blocks for the same striped block group, so that
{{countNodes}} can use this information to compute the #. I will use this jira
to fix this part and add more unit tests covering reporting bad blocks from
DFSClient to NN.
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
