[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938701#comment-14938701
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938509#comment-14938509
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938132#comment-14938132
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937912#comment-14937912
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937659#comment-14937659
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937409#comment-14937409
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8548/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-09-30 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937213#comment-14937213
 ] 

Hudson commented on HDFS-8543:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/])
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: 
rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java


> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-19 Thread Jing Zhao (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593775#comment-14593775
 ] 

Jing Zhao commented on HDFS-8543:
-

The test failures are caused by "java.lang.NoClassDefFoundError: 
org/apache/hadoop/hdfs/protocol/LocatedStripedBlock", thus should be unrelated. 
The latest patch looks good to me. I will commit it shortly.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587911#comment-14587911
 ] 

Hadoop QA commented on HDFS-8543:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m 38s | Findbugs (version ) appears to 
be broken on HDFS-7285. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 40s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 51s | There were no new javadoc 
warning messages. |
| {color:red}-1{color} | release audit |   0m 15s | The applied patch generated 
1 release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 38s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:red}-1{color} | findbugs |   3m 28s | The patch appears to introduce 4 
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m 21s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 100m 41s | Tests failed in hadoop-hdfs. |
| | | 143m 49s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
|   | hadoop.hdfs.TestRead |
|   | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
|   | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
|   | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
|   | hadoop.hdfs.TestHdfsAdmin |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
|   | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
|   | hadoop.hdfs.TestClientReportBadBlock |
|   | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
|   | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
|   | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
|   | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
|   | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.TestAppendSnapshotTruncate |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
|   | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
|   | 
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
|   | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
|   | hadoop.hdfs.TestFileAppendRestart |
|   | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
|   | hadoop.cli.TestErasureCodingCLI |
|   | hadoop.hdfs.protocol.TestBlockListAsLongs |
|   | hadoop.hdfs.server.namenode.TestHDFSConcat |
|   | hadoop.TestRefreshCallQueue |
|   | hadoop.hdfs.TestListFilesInDFS |
|   | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
|   | hadoop.hdfs.TestEncryptedTransfer |
|   | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
|   | hadoop.hdfs.TestMiniDFSCluster |
|   | hadoop.hdfs.server.mover.TestMover |
|   | hadoop.security.TestPermissionSymlinks |
|   | hadoop.hdfs.TestDFSRollback |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
|   | hadoop.hdfs.TestFileConcurrentReader |
|   | hadoop.hdfs.TestFileAppend2 |
|   | hadoop.hdfs.server.datanode.TestDataNodeExit |
|   | hadoop.hdfs.TestGetFileChecksum |
|   | hadoop.security.TestRefreshUserMappings |
|   | hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys |
|   | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
|   | hadoop.hdfs.TestRecoverStripedFile |
|   | hadoop.hdfs.server.namenode.TestAllowFormat |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
|   | hadoop.hdfs.server.namenode.TestDeadDatanode |
|   | hadoop.hdfs.server.namenode.TestAuditLogs |
|   | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
|   | hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy 
|
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
|   | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
|   | hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks |
|   | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
|   | hadoop.hdfs.TestBlockStoragePolicy |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
|   | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
|   | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
|   | hadoop.hdfs.TestCrcCorruption |
|   | hadoop.hdfs.server.blockmanagement.TestBlock

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-15 Thread Jing Zhao (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587106#comment-14587106
 ] 

Jing Zhao commented on HDFS-8543:
-

Thanks for working on this, Walter! So some comments other than Zhe's comments:
# One minor is that we can optimize the following code:
{code}
+// for each duplicated index, delete some replicas until only one left
+for (int targetIndex = duplicated.nextSetBit(0); targetIndex >= 0;
+ targetIndex = duplicated.nextSetBit(targetIndex + 1)) {
+  List candidates = new ArrayList<>();
+  for (DatanodeStorageInfo storage : nonExcess) {
+int index = sblk.getStorageBlockIndex(storage);
+if (index == targetIndex) {
+  candidates.add(storage);
+}
+  }
{code}
{{getStorageBlockIndex}} has to go through the whole object triplet array. 
Looks like we can avoid making this call in the inner loop by caching the 
mapping of storage and index into a temp map.
# The following code in {{chooseExcessReplicasStriped}} works currently but 
looks like the 2nd and 3rd parameters are not semantically consistent with the 
parameter {{candidates}}? {{candidates}} is for the internal block while the 
former two parameters are about the whole block group. Maybe we can use a new 
block object with the internal block's ID and replication factor 1 here.
{code}
DatanodeStorageInfo target = placementPolicy.chooseReplicaToDelete(bc,
storedBlock, groupSize, candidates, empty, excessTypes);
{code}

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585066#comment-14585066
 ] 

Hadoop QA commented on HDFS-8543:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m  9s | Findbugs (version ) appears to 
be broken on HDFS-7285. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 31s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 35s | There were no new javadoc 
warning messages. |
| {color:red}-1{color} | release audit |   0m 15s | The applied patch generated 
1 release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 37s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:red}-1{color} | findbugs |   3m 25s | The patch appears to introduce 4 
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 172m 16s | Tests failed in hadoop-hdfs. |
| | | 214m 20s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.hdfs.server.namenode.TestAuditLogs |
|   | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12739482/HDFS-8543-HDFS-7285.06.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 49d5cff |
| Release Audit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11345/artifact/patchprocess/patchReleaseAuditProblems.txt
 |
| Findbugs warnings | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11345/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
 |
| hadoop-hdfs test log | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11345/artifact/patchprocess/testrun_hadoop-hdfs.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11345/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11345/console |


This message was automatically generated.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, 
> HDFS-8543-HDFS-7285.06.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14584962#comment-14584962
 ] 

Hadoop QA commented on HDFS-8543:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m 23s | Findbugs (version ) appears to 
be broken on HDFS-7285. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 2 new or modified test files. |
| {color:green}+1{color} | javac |   7m 29s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 42s | There were no new javadoc 
warning messages. |
| {color:red}-1{color} | release audit |   0m 14s | The applied patch generated 
1 release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 38s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:red}-1{color} | findbugs |   3m 25s | The patch appears to introduce 4 
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 179m 10s | Tests failed in hadoop-hdfs. |
| | | 221m 29s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
|   | hadoop.hdfs.server.mover.TestMover |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
|   | hadoop.hdfs.server.namenode.TestAuditLogs |
|   | hadoop.hdfs.server.balancer.TestBalancer |
|   | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
|   | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12739464/HDFS-8543-HDFS-7285.05.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 683332b |
| Release Audit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11340/artifact/patchprocess/patchReleaseAuditProblems.txt
 |
| Findbugs warnings | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11340/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
 |
| hadoop-hdfs test log | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11340/artifact/patchprocess/testrun_hadoop-hdfs.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11340/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11340/console |


This message was automatically generated.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-11 Thread Zhe Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582275#comment-14582275
 ] 

Zhe Zhang commented on HDFS-8543:
-

Thanks Walter for the update. No worries :) The patch LGTM after addressing the 
following issues:

# Move {{=}} to previous line
{code}
+final DatanodeStorageInfo delStorageHint
+= DatanodeStorageInfo.getDatanodeStorageInfo(nonExcess, delNodeHint);
{code}
# We can probably use something like the below to simplify the {{while 
(!duplicated.isEmpty())}} loop (from a quick check of the documentation of 
{{BitSet}}):
{code}
 for (int i = bs.nextSetBit(0); i >= 0; i = bs.nextSetBit(i+1)) {
 // operate on index i here
 }
 {code}
 # What if {{delHint}} points to an internal block with many excess replicas? 
Should we check before clearing it from {{duplicated}}?
 {code}
 +  if (duplicated.get(index)) {
 +processChosenExcessReplica(nonExcess, delStorageHint, storedBlock);
 +duplicated.clear(index);
 +  }
 {code}
 # Nit in the test: let's assign a type for the {{HashSet}}
 {code}
 HashSet indices = new HashSet();
{code}
# Not necessarily in this JIRA, but we can test different excess patterns, at 
least we should cover the case where an internal block has 3 copies.
 # Looks like the changes in {{CHANGES-HDFS-EC-7285.txt}} and the xml file is 
unrelated.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-10 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581513#comment-14581513
 ] 

Hadoop QA commented on HDFS-8543:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  0s | The patch command could not apply 
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12738992/HDFS-8543-HDFS-7285.04.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 98d3407 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11309/console |


This message was automatically generated.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, 
> HDFS-8543-HDFS-7285.04.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-10 Thread Zhe Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581465#comment-14581465
 ] 

Zhe Zhang commented on HDFS-8543:
-

Thanks for the update Walter.
{code}
  used.set(index);
  if (used.get(index)) {
{code}
Looks like the {{if}} will always hit {{true}}? Could you add some comments for 
{{used}} and {{duplicated}?

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-10 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581456#comment-14581456
 ] 

Hadoop QA commented on HDFS-8543:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m 15s | Findbugs (version ) appears to 
be broken on HDFS-7285. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 1 new or modified test files. |
| {color:red}-1{color} | javac |   7m 34s | The applied patch generated  1  
additional warning messages. |
| {color:green}+1{color} | javadoc |   9m 50s | There were no new javadoc 
warning messages. |
| {color:red}-1{color} | release audit |   0m 14s | The applied patch generated 
1 release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 39s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:red}-1{color} | findbugs |   3m 26s | The patch appears to introduce 4 
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 173m  9s | Tests failed in hadoop-hdfs. |
| | | 215m 36s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
|   | hadoop.cli.TestErasureCodingCLI |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
|   | hadoop.hdfs.server.namenode.TestAuditLogs |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.TestEncryptedTransfer |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12738961/HDFS-8543-HDFS-7285.03.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / c5d4652 |
| javac | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11307/artifact/patchprocess/diffJavacWarnings.txt
 |
| Release Audit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11307/artifact/patchprocess/patchReleaseAuditProblems.txt
 |
| Findbugs warnings | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11307/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
 |
| hadoop-hdfs test log | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11307/artifact/patchprocess/testrun_hadoop-hdfs.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11307/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11307/console |


This message was automatically generated.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch, 
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-10 Thread Zhe Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581265#comment-14581265
 ] 

Zhe Zhang commented on HDFS-8543:
-

Thanks Walter for working on this! The main logic of 
{{chooseExcessReplicates4Stripe}} looks reasonable.

# The name {{ExcessReplicates}} should be {{ExcessReplicas}}. It was originated 
in trunk. I will file a separate JIRA to address.
# We could assert the type before casting:
{code}
BlockInfoStriped sblk = (BlockInfoStriped) storedBlock;
{code}
# It seems no value is ever inserted into {{used}}?
# {{dulplicated}} misspelled
# The 2 versions of {{chooseExcessReplicates}} should be _contiguous_ and 
_striped_, instead of _replication_ and _stripe_.

Will review again when above issues are addressed, especially #3.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block

2015-06-08 Thread Walter Su (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578160#comment-14578160
 ] 

Walter Su commented on HDFS-8543:
-

The problem of the old code: Assume blockGroup=\[0, 1, 2, 3\]. We lose idx=2, 
and are left \[0, 1, 3\]. After recovery, we get\[0, 1, 2, 3\]. Assume dead DN 
comes back. We have\[0, 1, 2, 3, 2\]. The old processOverReplicatedBlock() may 
delete idx=1, so we are left \[0, 2, 3, 2\], we have to recover it again. It 
could be an endless loop.

> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2. 
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)