[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937183#comment-14937183
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937380#comment-14937380
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8548/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937629#comment-14937629
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937883#comment-14937883
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938103#comment-14938103
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938480#comment-14938480
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938672#comment-14938672
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288561#comment-14288561
]
Tsz Wo Nicholas Sze commented on HDFS-7652:
---
We may be able to considers a EC block as regular block. In that case, no
change is needed for block report. Will post more details later.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289879#comment-14289879
]
Zhe Zhang commented on HDFS-7652:
-
This design question is mainly [discussed |
https://issues.apache.org/jira/browse/HDFS-7339?focusedCommentId=14289868&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14289868]
under HDFS-7339
Again, I really appreciate the in-depth thoughts! [~szetszwo] [~jingzhao]
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305670#comment-14305670
]
Zhe Zhang commented on HDFS-7652:
-
This JIRA can only be tested with client striped writing.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310154#comment-14310154
]
Jing Zhao commented on HDFS-7652:
-
Thanks for working on this, Zhe! Basically to process a report for a stripped
block we always need these two pieces of information: 1) the reported block
that contains the block id seen by DN, and 2) the corresponding
BlockInfo(Stripped) object stored in the blocksMap. The former is necessary to
identify the internal index of the reported block inside of the block group.
This part of logic has been included in the current patch in HDFS-7716. For
this jira, I think {{BlockManager#getStoredBlock}} should be the place to add
the logic identifying if the reported block points to a contiguous block or a
stripped block group (based on the id and also map lookup result).
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310406#comment-14310406
]
Zhe Zhang commented on HDFS-7652:
-
Good points Jing! I've updated the patch incorporating these changes.
bq. I'd like to keep the blocksMap as a pure data structure storing block and
blockInfo thus I still move the logic into BlockManager
Agreed.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310558#comment-14310558
]
Zhe Zhang commented on HDFS-7652:
-
[~jingzhao] Your implementation of {{getStoredBlock}} in 003 patch is correct
-- it handles conflicts with legacy (randomly generated) block IDs. I just
updated the patch.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311631#comment-14311631
]
Jing Zhao commented on HDFS-7652:
-
Thanks Zhe for updating the patch. The patch now needs a small rebase. Besides,
looks like the new unit test will not pass since we no longer have the changing
block id logic. I suggest removing this test and the
{{changeBlockIDForTesting}} method from the patch. We can add more tests later
in/after HDFS-7716.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312528#comment-14312528
]
Zhe Zhang commented on HDFS-7652:
-
Thanks Jing for taking care of the {{BlockInfo}} refactor! I agree about
keeping this patch small and fill in more details after the {{BlockInfo}}
changes are finalized.
Attached a smaller patch which is basically the same as 003 except for the
small rebase for HDFS-7743. Now {{getStoredBlock}} always returns
{{BlockInfoContiguous}} and let's change it with HDFS-7716.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312566#comment-14312566
]
Jing Zhao commented on HDFS-7652:
-
Thanks Zhe! +1 for the latest patch.
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312575#comment-14312575
]
Zhe Zhang commented on HDFS-7652:
-
I just committed to HDFS-EC. Thanks Jing for the help on this JIRA!
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
