[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185052#comment-15185052 ] Rakesh R commented on HDFS-8786: Attached another patch fixing checkstyle comments. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-005.patch, > HDFS-8786-006.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184813#comment-15184813
]
Hadoop QA commented on HDFS-8786:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
142 unchanged - 1 fixed = 144 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 58s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 51s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.TestHFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791964/HDFS-8786-005.patch |
| JIRA Issue | HDFS-8786 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184708#comment-15184708 ] Rakesh R commented on HDFS-8786: Thanks [~jingzhao], I've attached patch fixing other comments. Please review it again when you get a chance. Raised HDFS-9918 for the sorting logic. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-005.patch, > HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184419#comment-15184419 ] Jing Zhao commented on HDFS-8786: - Thanks [~rakeshr]! For #1 I think I misread your patch so please ignore my previous comment. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184373#comment-15184373
]
Rakesh R commented on HDFS-8786:
Thanks [~jingzhao] for the reviews and good comments.
Could you please give few clarifications on first comment. I will take care
other comments while preparing next patch. Thanks!
comment-2=> agreed, will update in next patch
comment-3=> agreed, will raise a follow-on jira and work on.
comment-4=> agreed, will update in next patch
bq. comment-1=> 1. Not all the decommissioning nodes can be later used as
source nodes, since we still need to consider DataNode's current load etc. Thus
I'm not sure the calculation is correct here. In the meanwhile, I do not think
we should adjust additionalReplRequired: there is no need to leave
decommissioning nodes to the next round. Thus looks like we do not need this
change.
{{getAdditionalReplRequired()}} count/value is used while choosing the required
number of target nodes logic as shown below and these target nodes will be
given to reconstruction or replication tasks. The new calculation in my patch
is adjusting the additional replication required by ignoring the
decommissioning count, so that the target nodes for these will not be chosen
later. Here the idea is, will schedule decommissioning replication task only if
there are no other cases like, under replcias or needed a rack etc. Again
ErasureCodingWork will schedule replication task for decommissioning only if
the block group has {{#hasAllInternalBlocks}}. IIUC correctly your [previous
comment|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15174447&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15174447]
also pointing to this ignore the decommssioning if any other tasks exists. I
think the existing {{BlockManager#chooseSourceDatanodes}} is sufficient for
source selection, am I missing anything?
{code}
case-1) 9 live replicas on 5 racks and 1 decommissioning replica
Choose 1 target for rack replication task and ignore decommissioning
replication task
case-2) 7 live replicas, 1 under replica and 1 decommissioning replica
Choose 1 target for reconstruction task and ignore decommissioning replication
task
case-3) 6 live replicas, 2 under replica and 1 decommissioning replica
Choose 2 target for reconstruction task and ignore decommissioning replication
task
case-4) 8 live replicas and 1 decommissioning replica
Choose 1 target for decommissioning replication task
case-5) 7 live replicas and 2 decommissioning replica
Choose 2 target for decommissioning replication task
{code}
{code}
ErasureCodingWork.java:-
void chooseTargets(BlockPlacementPolicy blockplacement,
BlockStoragePolicySuite storagePolicySuite,
Set excludedNodes) {
//
DatanodeStorageInfo[] chosenTargets = blockplacement.chooseTarget(
getBc().getName(), getAdditionalReplRequired(), getSrcNodes()[0],
getLiveReplicaStorages(), false, excludedNodes,
getBlock().getNumBytes(),
storagePolicySuite.getPolicy(getBc().getStoragePolicyID()));
ReplicationWork.java:-
void chooseTargets(BlockPlacementPolicy blockplacement,
BlockStoragePolicySuite storagePolicySuite,
Set excludedNodes) {
//
DatanodeStorageInfo[] chosenTargets = blockplacement.chooseTarget(
getBc().getName(), getAdditionalReplRequired(), getSrcNodes()[0],
getLiveReplicaStorages(), false, excludedNodes,
getBlock().getNumBytes(),
storagePolicySuite.getPolicy(getBc().getStoragePolicyID()));
{code}
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch,
> HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183810#comment-15183810
]
Jing Zhao commented on HDFS-8786:
-
Thanks for updating the patch, [~rakeshr]! The 004 patch looks pretty good to
me. Some minors:
# Not all the decommissioning nodes can be later used as source nodes, since we
still need to consider DataNode's current load etc. Thus I'm not sure the
calculation is correct here. In the meanwhile, I do not think we should adjust
additionalReplRequired: there is no need to leave decommissioning nodes to the
next round. Thus looks like we do not need this change.
{code}
// should reconstruct all the internal blocks before scheduling
// replication task for decommissioning node(s).
if (additionalReplRequired - numReplicas.decommissioning() > 0) {
additionalReplRequired = additionalReplRequired
- numReplicas.decommissioning();
}
{code}
# We actually need to track if the reconstruction work is triggered only by
"not enough rack". For normal EC reconstruction work maybe we also do not have
enough racks, but we will keep {{enoughRack}} as true. Thus using a boolean
"notEnoughRack" may be more accurate.
{code}
private boolean enoughRack = true;
{code}
# {{DatanodeManager#sortLocatedStripedBlocks}} can be added in a separate jira.
We can also add some new tests for this change.
# In ErasureCodingWork, We can put the not-enough-rack logic and the
decommissioning logic into two separate methods. Also if target size is smaller
than sources, we do not need to create the block in {{addTaskToDatanode}}.
Maybe the code after change can look like this:
{code}
private void createReplicationWork(int sourceIndex, DatanodeStorageInfo
target) {
BlockInfoStriped stripedBlk = (BlockInfoStriped) getBlock();
final byte blockIndex = liveBlockIndicies[sourceIndex];
final DatanodeDescriptor source = getSrcNodes()[sourceIndex];
final long internBlkLen = StripedBlockUtil.getInternalBlockLength(
stripedBlk.getNumBytes(), stripedBlk.getCellSize(),
stripedBlk.getDataBlockNum(), blockIndex);
final Block targetBlk = new Block(
stripedBlk.getBlockId() + blockIndex, internBlkLen,
stripedBlk.getGenerationStamp());
source.addBlockToBeReplicated(targetBlk, new DatanodeStorageInfo[]{target});
if (BlockManager.LOG.isDebugEnabled()) {
BlockManager.LOG.debug("Add replication task from source {} to "
+ "target {} for EC block {}", source, target, targetBlk);
}
}
private List findDecommissioningSources() {
List srcIndices = new ArrayList<>();
for (int i = 0; i < getSrcNodes().length; i++) {
if (getSrcNodes()[i].isDecommissionInProgress()) {
srcIndices.add(i);
}
}
return srcIndices;
}
@Override
void addTaskToDatanode(NumberReplicas numberReplicas) {
final DatanodeStorageInfo[] targets = getTargets();
assert targets.length > 0;
BlockInfoStriped stripedBlk = (BlockInfoStriped) getBlock();
if (hasNotEnoughRack()) {
// if we already have all the internal blocks, but not enough racks,
// we only need to replicate one internal block to a new rack
int sourceIndex = chooseSource4SimpleReplication();
createReplicationWork(sourceIndex, targets[0]);
} else if (numberReplicas.decommissioning() > 0 && hasAllInternalBlocks()) {
List decommissioningSources = findDecommissioningSources();
// decommissioningSources.size() should be >= targets.length
final int num = Math.min(decommissioningSources.size(), targets.length);
for (int i = 0; i < num; i++) {
createReplicationWork(decommissioningSources.get(i), targets[i]);
}
} else {
targets[0].getDatanodeDescriptor().addBlockToBeErasureCoded(
new ExtendedBlock(blockPoolId, stripedBlk),
getSrcNodes(), targets, getLiveBlockIndicies(),
stripedBlk.getErasureCodingPolicy());
}
}
{code}
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch,
> HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181564#comment-15181564
]
Hadoop QA commented on HDFS-8786:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 19s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hdfs.server.namenode.TestEditLog |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.server.namenode.TestSecureNameNode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791599/HDFS-8786-004.patch |
| JIRA Issue | HDFS-8786 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b8204b7ebd9c 3.13.0-36-lowlatency #63-Ubuntu SMP PRE
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181501#comment-15181501 ] Rakesh R commented on HDFS-8786: Thanks a lot [~jingzhao] for the detailed explanations and advice. Sorry for the delay, I have tried an attempt to fix both the comments and attached patch for the same. I agree to split the task based on the latest patch feedback. Kindly review the patch again. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180415#comment-15180415 ] Jing Zhao commented on HDFS-8786: - I think we can use this jira to fix ErasureCodingWork first. The sorting logic can be done as a follow-on. Please let me know if you have further questions, [~rakeshr]. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-003.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174467#comment-15174467
]
Jing Zhao commented on HDFS-8786:
-
To avoid the changes inside of BlockInfoStriped, I think we can do the
following: since we always use two arrays (one for datanodes and the other for
their internal block indices) when returning a striped block to client, we can
adjust the order inside of the LocatedStripedBlock instead of BlockInfoStriped.
I.e., the order adjustment (based on decommissioning DN information) should be
done against LocatedStripedBlock in the same step as {{sortLocatedBlocks}}.
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch,
> HDFS-8786-003.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174447#comment-15174447
]
Jing Zhao commented on HDFS-8786:
-
Thanks for updating the patch, [~rakeshr]. Comments on the current patch:
1. For ErasureCodingWork, we have to handle the following scenarios when
{{hasAllInternalBlocks}} returns true:
#* we have decommissioning DN
#* we have enough DN but not enough racks
#* the above two situations happen at the same time
Things may get a little complicated when decommissioning situation and more
racks situation get mixed. For example, it is possible that there are 9 live
internal blocks on 5 racks, and 1 more internal block in a decommissioning
datanode. In this situation, we will only choose 1 target and the
decommissioning dn should be ignored. In another example, if we have 8 live
replicas and 1 decommissioning replica, we should replicate the decommissioning
replica. Looks to me the current patch cannot handle all the scenarios.
Currently I think we should explicitly let ErasureCodingWork know if the
reconstruction work is triggered by not-enough-racks. We can have this check in
{{validateReconstructionWork}}, and pass the result into the ErasureCodingWork
instance. Later when adding task to DN, we should first check this result, and
if it is true, run the current code added by HDFS-9818. If it is false, we
check if the source nodes cover all the internal blocks but contain
decommissioning datanode, and schedule replication work for it if necessary.
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch,
> HDFS-8786-003.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169280#comment-15169280
]
Hadoop QA commented on HDFS-8786:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 18s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12790137/HDFS-8786-003.patch |
| JIRA Issue | HDFS-8786 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 97aabce3dbc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_6
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169077#comment-15169077 ] Rakesh R commented on HDFS-8786: Note: Attached another patch to fix unit test failure. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-003.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168889#comment-15168889
]
Hadoop QA commented on HDFS-8786:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 47s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 32s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestDecommissionWithStriped |
| JDK v1.7.0_95 Failed junit tests |
hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs |
| | hadoop.hdfs.TestDecommissionWithStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12790097/HDFS-8786-002.patch |
| JIRA Issue | HDFS-8786 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Lin
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168765#comment-15168765 ] Rakesh R commented on HDFS-8786: Thanks a lot [~jingzhao] for the explanation. I've attached another patch to show the changes. I hope I've covered your first comment. But for the second one I still kept the changes inside BlockStripedInfo class because this logic requires block indices references to do the switching. Do you have any suggestion/advice to me. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, > HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167762#comment-15167762
]
Jing Zhao commented on HDFS-8786:
-
bq. I will add an extra check against the BlockManager#corruptReplicas map. If
true will trigger EC reconstruction task otherwise replication task.
corruptReplicas is not tracking if the whole blockInfo (i.e., the whole striped
block group) is corrupted or requires internal block reconstructions. It is
only used to track corrupted internal blocks/replicas that were reported by
DataNode/client by detected through block reports. Thus here the key is not to
check corruptReplicas, but to check if we have enough healthy internal blocks
covering the complete ID range (i.e. {{hasAllInternalBlocks}}). I.e., we should
augment the current {{chooseSource4SimpleReplication}} method by adding
choosing decommissioning node logic.
For sorting the storages, we do not do real "sorting". Instead, we only need to
check if we have duplicated internal blocks in the block group, and if yes, we
make sure a decommissioned storage with duplicated block is put in the end.
I.e., suppose we have storages
d0, d1, d2, d3, d4, d5, d6, d7, d8, d9
mapping to indices
0, 1, 2, 3, 4, 5, 6, 7, 8, 2
Thus we have duplicated internal blocks for b2, locating in d2 and d9. If we
find that d2 is a decommissioned node and d9 is not, we should switch d2 and d9
in the storage list.
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167188#comment-15167188
]
Rakesh R commented on HDFS-8786:
Thanks [~jingzhao] for the useful comments.
bq. We can create replication task for an EC block only when we can make sure
we have all the internal blocks (the source nodes can be in
decommissioned/decommissioning state). Otherwise even if we have
decommissioned/decommissioning datanodes, we should still trigger an EC
reconstruction task.
I will add an extra check against the {{BlockManager#corruptReplicas}} map. If
true will trigger EC reconstruction task otherwise replication task. OK?
bq. For changes in BlockInfoStriped, instead of directly mixing decommission
related logic inside, it will be better to add a sorting logic in BlockManager
just as what we do for contiguous blocks.
I think you are suggesting to provide a comparator similar to
{{DFSUtil.DECOM_COMPARATOR}}. Just adding a thought to understand more on this
change -> I could see for a (m+k) block group, (m+k) storage units are sorted.
say (0,1,2,3,4). With the current behavior it will add new storage unit at the
end of the list (0,1,2,3,4,0) and later after the removal of decommission node,
the indices list will become (1,2,3,4,5,0). Will this break the sorted order
behavior? Please correct me if I'm missing anything. Thanks!
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159615#comment-15159615
]
Jing Zhao commented on HDFS-8786:
-
Thanks for working on this, Rakesh. Some quick comments:
# We can create replication task for an EC block only when we can make sure we
have all the internal blocks (the source nodes can be in
decommissioned/decommissioning state). Otherwise even if we have
decommissioned/decommissioning datanodes, we should still trigger an EC
reconstruction task.
# For changes in {{BlockInfoStriped}}, instead of directly mixing decommission
related logic inside, it will be better to add a sorting logic in BlockManager
just as what we do for contiguous blocks.
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158502#comment-15158502 ] Rakesh R commented on HDFS-8786: Thank you [~drankye] for the help. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158296#comment-15158296 ] Kai Zheng commented on HDFS-8786: - Thanks [~rakeshr] for working on this. I will take a look at the patch. > Erasure coding: DataNode should transfer striped blocks before being > decommissioned > --- > > Key: HDFS-8786 > URL: https://issues.apache.org/jira/browse/HDFS-8786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Rakesh R > Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch > > > Per [discussion | > https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004] > under HDFS-8697, it's too expensive to reconstruct block groups for decomm > purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158183#comment-15158183
]
Rakesh R commented on HDFS-8786:
Attached patch to support decommissioning node which contains striped blocks.
Could you please review the patch when you get a chance. Thanks!
Following are the major changes:
- BlockManager will schedule data_transfer task to copy the block from
decommissioning src node to a target node.
- While adding new node BlockInfoStriped will keep the new node to the old
decommission node blkindex and move the old storage to the end of the
{{storages}} list. Later this decommission node index will get removed.
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157396#comment-15157396
]
Hadoop QA commented on HDFS-8786:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 12s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 24s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 197m 19s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| JDK v1.7.0_95 Failed junit tests |
hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12789015/HDFS-8786-001.patch |
| JIRA Issue | HDFS-8786 |
| Optional Tests | as
[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150908#comment-15150908
]
Rakesh R commented on HDFS-8786:
I’ve started analyzing this and attached a draft patch for discussion.
Currently, the striped blocks from the decommissioned node is considered as
insufficiently replicated and is sending EC_RECONSTRUCT command to the
datanode. But I’ve below observation and requires changes in this part. It
would be helpful to get input from others. Thanks!
# During reconstruction process, like contiguous block behavior the
decommissioning datanode also be added to the {{liveBlockIndices}} list, which
will trouble the reconstruction decoding algo and throws the following
exception. In the draft patch, I just skipped the source node for simplicity.
Can we consider decommission as a special case and skip the erased index checks
? It would be good if someone can give more details about this condition.
{code}
RSRawDecoder.java
//
if (erasedIndexes[i] == erasedOrNotToReadIndexes[j]){
found = true;
}
//
if (!found) {
throw new HadoopIllegalArgumentException(
"Inputs not fully corresponding to erasedIndexes in null places");
}
{code}
# Secondly, contiguous block have the sorting logic and is keeping the
decommissioned node at the end of the block location list. This is done using
{{DFSUtil.DECOM_COMPARATOR}} as shown below. But in the striped block, it
doesn’t have the sorting logic and is maintaining the index position. Whats
your opinion in replacing the decommissioned storage index with the newly
reconstructed node and keep the decommissioned node at the end in
{{BlockInfoStriped}} itself.
{code}
DatanodeManager.java
Comparator comparator = avoidStaleDataNodesForRead ?
new DFSUtil.DecomStaleComparator(staleInterval) :
DFSUtil.DECOM_COMPARATOR;
{code}
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> ---
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
