[
https://issues.apache.org/jira/browse/HDFS-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299116#comment-14299116
]
Jing Zhao commented on HDFS-7339:
---------------------------------
Thanks Zhe! The patch looks good overall. Some comments and questions:
# Instead of the current ID division mechanism (calculating the mid point
between LAST_RESERVED_BLOCK_ID and LONG.MAX), can we simply let the block group
id take all the negative long space (i.e., with first bit set to 1)? In this
way we can utilize larger space and use simple bit manipulations for id
generation/checking.
# Why do we need to reserve the first 1024 block group ids?
# If we directly extend the current BlockInfo to BlockGroupInfo, the semantic
of the {{triplets}} may be different for BlockGroupInfo. One possible solution
is to let {{triplets}}'s size be {{3*(k+m)}}, where k is the number of data
blocks and m is the number of the parity blocks.
# The current BlockGroupInfo's constructor calls BlockInfo's copy constructor
which constructs triplets based on replication factor. We may still need to
revisit BlockInfo and BlockGroupInfo to make sure BlockGroupInfo is strictly
separated with replication operations and logic.
The above #3 and #4 may need some extra refactoring work on the current
BlockInfo class. I'm also fine with moving this part of work to a separate jira.
> Allocating and persisting block groups in NameNode
> --------------------------------------------------
>
> Key: HDFS-7339
> URL: https://issues.apache.org/jira/browse/HDFS-7339
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-7339-001.patch, HDFS-7339-002.patch,
> HDFS-7339-003.patch, HDFS-7339-004.patch, HDFS-7339-005.patch,
> HDFS-7339-006.patch, HDFS-7339-007.patch, Meta-striping.jpg, NN-stripping.jpg
>
>
> All erasure codec operations center around the concept of _block group_; they
> are formed in initial encoding and looked up in recoveries and conversions. A
> lightweight class {{BlockGroup}} is created to record the original and parity
> blocks in a coding group, as well as a pointer to the codec schema (pluggable
> codec schemas will be supported in HDFS-7337). With the striping layout, the
> HDFS client needs to operate on all blocks in a {{BlockGroup}} concurrently.
> Therefore we propose to extend a file’s inode to switch between _contiguous_
> and _striping_ modes, with the current mode recorded in a binary flag. An
> array of BlockGroups (or BlockGroup IDs) is added, which remains empty for
> “traditional” HDFS files with contiguous block layout.
> The NameNode creates and maintains {{BlockGroup}} instances through the new
> {{ECManager}} component; the attached figure has an illustration of the
> architecture. As a simple example, when a {_Striping+EC_} file is created and
> written to, it will serve requests from the client to allocate new
> {{BlockGroups}} and store them under the {{INodeFile}}. In the current phase,
> {{BlockGroups}} are allocated both in initial online encoding and in the
> conversion from replication to EC. {{ECManager}} also facilitates the lookup
> of {{BlockGroup}} information for block recovery work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
