[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937878#comment-14937878 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938097#comment-14938097 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938475#comment-14938475 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938667#comment-14938667 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937623#comment-14937623 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937375#comment-14937375 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937177#comment-14937177 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570459#comment-14570459
]
Hadoop QA commented on HDFS-8328:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 8s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 45s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 14s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 37s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 29 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 23s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 169m 18s | Tests failed in hadoop-hdfs. |
| | | 211m 28s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.util.TestStripedBlockUtil |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737083/HDFS-8328-HDFS-7285.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 5f15084 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11217/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11217/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11217/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11217/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11217/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11217/console |
This message was automatically generated.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572058#comment-14572058
]
Zhe Zhang commented on HDFS-8328:
-
Thanks Yi for the update (and for this great work on DN!). The latest patch
looks good to me; +1 on committing it
Just as a future reference for consolidation with client side striped reading
code:
# We can consider moving {{StripeReader}} from HDFS-8319 to the util class and
subclass it for DN
# {{readMinimumStripedData4Recovery}} is the counterpart of
{{StripeReader#readStripe}}
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568816#comment-14568816
]
Walter Su commented on HDFS-8328:
-
1a. The first while loop should guarded with {{!used.get(m)}}. m starting from
{{stripedReaders.size()}} doesn't mean it's not used.
{code}
private int scheduleNewRead(BitSet used) {
StripedReader reader = null;
int m = stripedReaders.size();
while (reader == null m sources.length) {
if(!used.get(m)) {
reader = addStripedReader(m, positionInBlock);
if (getReadLength(liveIndices[m]) 0) {
if (reader.blockReader == null) {
reader = null;
}
}
}
m++;
}
{code}
1b. if getReadLength(liveIndices\[m\]) == 0, we only have to keep one
{{return}} statement at second for loop.
2.paddingBufferToLen(..) for multiple zeroStripeBuffers[] is wasting time. One
zeroStripeBuffer is enough.
{code}
private void recoverTargets(int[] success, boolean[] targetsStatus,
...
if (success.length dataBlkNum) {
for (int i = 0; i zeroStripeBuffers.length; i++) {
ByteBuffer buffer = zeroStripeBuffers[i];
paddingBufferToLen(buffer, toRecoverLen);
int index = covertIndex4Decode(zeroStripeIndices[i], dataBlkNum,
parityBlkNum);
inputs[index] = (ByteBuffer)buffer.flip();
}
{code}
We prepare one zeroStripeBuffer. Then use ByteBuffer.slice() to create a
mirror buffer. It saves memory and time.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569817#comment-14569817 ] Zhe Zhang commented on HDFS-8328: - bq. paddingBufferToLen(..) for multiple zeroStripeBuffers[] is wasting time. One zeroStripeBuffer is enough. Good thought Walter. But as Kai mentioned on another JIRA, the input buffers could be altered in the decode method. Maybe we should do this zero copy update after the HADOOP side changes to keep input buffers unchanged. Follow-on to update decode for DataNode striped blocks reconstruction - Key: HDFS-8328 URL: https://issues.apache.org/jira/browse/HDFS-8328 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Attachments: HDFS-8328-HDFS-7285.001.patch, HDFS-8328-HDFS-7285.002.patch Current the decode for DataNode striped blocks reconstruction is a workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570204#comment-14570204 ] Walter Su commented on HDFS-8328: - I see. The logic of scheduleNewRead(..) is correct. Follow-on to update decode for DataNode striped blocks reconstruction - Key: HDFS-8328 URL: https://issues.apache.org/jira/browse/HDFS-8328 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Attachments: HDFS-8328-HDFS-7285.001.patch, HDFS-8328-HDFS-7285.002.patch Current the decode for DataNode striped blocks reconstruction is a workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570199#comment-14570199 ] Kai Zheng commented on HDFS-8328: - Yi HDFS-8517 borrowed the utility function and had got committed, so you need a re-base. Thanks. Follow-on to update decode for DataNode striped blocks reconstruction - Key: HDFS-8328 URL: https://issues.apache.org/jira/browse/HDFS-8328 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Attachments: HDFS-8328-HDFS-7285.001.patch, HDFS-8328-HDFS-7285.002.patch Current the decode for DataNode striped blocks reconstruction is a workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570144#comment-14570144
]
Yi Liu commented on HDFS-8328:
--
Thanks for Zhe, Walter and Kai's comments. My reply to you guys:
--- *To Zhe:* --
{quote}
If initial connection to a target DN (chosen by NN) is unsuccessful,
getErasedIndices will not add the index in erasedIndices; therefore the block
will not be reconstructed. Should we send an ack to NN so the block can come
back to UnderReplicatedBlocks?
{quote}
This is a good question. I think no need currently. Same as continuous block
replication, if the replication failed, DN doesn't send an ack to NN. So,
yes, the block replica is already removed from {{neededReplications}}, next
time if some client request the block replica, client will report bad block
replica to NN, so the corrupted block replica is added back to
{{neededReplications}} again. I'd like to keep the same behavior.
{quote}
In each iteration of while (positionInBlock firstStripedBlockLength),
targetsStatus is reset to reflect if the transfer to each target is successful.
So when an element is flipped from true to false, it will never be flipped back.
{quote}
Actually {{targetsStatus}} is *not* reset in each iteration.
{{targetsStatus}} will record any failed target once, if any target failed, we
will never transfer data to it any more. Let me add more doc.
{quote}
zeroStripeBuffers is created based on whether the entire internal block is
empty. Ideally we should have such an array in each iteration of while
(positionInBlock firstStripedBlockLength) so we can avoid issuing read
requests as much as possible. I guess this can be a TODO for follow-on.
{quote}
I also thought we should try to find the empty internal block first, then less
read required, but only data block can be empty, and in most cases, data block
will be tried first. This is still an improvement. Agree to add a TODO and do
it in a follow-on.
{quote}
The updated {{scheduleNewRead}} is not so easy to follow.
{quote}
We have talked about this in the offline meeting:) Let me add more doc.
{quote}
The argument and return value of readMinimumStripedData4Recovery are not
trivial to understand. Maybe add them to the Javadoc?
{quote}
Also will add more doc about it.
{quote}
Since we are fetching mostly fixed size chunks, it's ideal to reuse buffers
instead of allocating new ones in addStripedReader. Walter Su has a good
analysis under HDFS-8481 on the GC issue.
{quote}
Actually stripedReader will be allocated only once, so the buffers are reused,
so there is no GC issue. Suppose the coder schema is 6+3, and 2 corrupted
striped blocks, suppose source DNs number is 7, target DNs number 2.If all
the source are valid, we will only allocate minimum(6) buffers and reuse them,
and the worst situation is we allocate 7 buffers and reuse them. I think it's
OK, of course, actually we only need to allocate 6 buffers in any situation,
this is a minor improvement, let me do it together with above .
--- *To Walter:* --
{quote}
The first while loop should guarded with !used.get(m). m starting from
stripedReaders.size() doesn't mean it's not used.
{quote}
Same as reply to Zhe, I think I need to add more doc here :)stripedReader
will be allocated only once, so current logic is correct.
{quote}
paddingBufferToLen(..) for multiple zeroStripeBuffers[] is wasting time. One
zeroStripeBuffer is enough.
{quote}
I ever considered this too, the reason I think Zhe replied you.
--- *To Kai:* --
{quote}
Noticed a typo: covertIndex4Decode should be convertIndex4Decode.
{quote}
Right, will update it.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569095#comment-14569095
]
Kai Zheng commented on HDFS-8328:
-
Noticed a typo: {{covertIndex4Decode}} should be convertIndex4Decode.
I borrowed the codes in HDFS-8517.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567347#comment-14567347
]
Hadoop QA commented on HDFS-8328:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 17s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 14s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 39s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 19 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 27s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 20s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 0m 25s | Tests failed in hadoop-hdfs. |
| | | 43m 29s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed build | hadoop-hdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736552/HDFS-8328-HDFS-7285.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 1299357 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11185/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11185/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11185/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11185/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11185/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11185/console |
This message was automatically generated.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568107#comment-14568107
]
Zhe Zhang commented on HDFS-8328:
-
Thanks for the update Yi! The patch looks good overall. I have the following
questions:
# The use of {{targetsStatus}} is a little tricky. Let me know if I'm
understanding it correctly:
#* If initial connection to a target DN (chosen by NN) is unsuccessful,
{{getErasedIndices}} will not add the index in {{erasedIndices}}; therefore the
block will not be reconstructed. Should we send an ack to NN so the block can
come back to {{UnderReplicatedBlocks}}?
#* In each iteration of {{while (positionInBlock firstStripedBlockLength)}},
{{targetsStatus}} is reset to reflect if the transfer to each target is
successful. So when an element is flipped from {{true}} to {{false}}, it will
never be flipped back.
#* If the above is correct, maybe let's add some Javadoc and TODO if necessary
(for the NN ack part)?
# {{zeroStripeBuffers}} is created based on whether the entire internal block
is empty. Ideally we should have such an array in each iteration of {{while
(positionInBlock firstStripedBlockLength)}} so we can avoid issuing read
requests as much as possible. I guess this can be a TODO for follow-on.
# The updated {{scheduleNewRead}} is not so easy to follow.
#* In the first loop, is it assumed that the reader indices in
{{stripedReaders}} begin from 0 and are consecutive?
{code}
int m = stripedReaders.size();
while (reader == null m sources.length) {
reader = addStripedReader(m, positionInBlock);
xxx
{code}
#* I feel the method does the right thing but haven't fully understood it yet.
It would be very helpful to have comments for steps 1, 2, 3 in the method.
# The argument and return value of {{readMinimumStripedData4Recovery}} are not
trivial to understand. Maybe add them to the Javadoc?
# Since we are fetching mostly fixed size chunks, it's ideal to reuse buffers
instead of allocating new ones in {{addStripedReader}}. [~walter.k.su] has a
good [analysis |
https://issues.apache.org/jira/browse/HDFS-8481?focusedCommentId=14564053page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14564053]
under HDFS-8481 on the GC issue.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch,
HDFS-8328-HDFS-7285.002.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564392#comment-14564392
]
Yi Liu commented on HDFS-8328:
--
Thanks Kai for comments.
{quote}
{{minRequiredSources}} looks like a little confusing, because from coder's
point of view
{quote}
This is not from coder's point of view, it's unrelated to coder. sources means
datanodes which contain correct striped block.
{quote}
How about renaming nullInputBuffers to nullCellBuffers
{quote}
Currently in DN, decode buffer size is not same as cell size. I have comment
{{// striped block length is 0}}, maybe I can change it to {{//The buffers and
indices for striped blocks whose length is 0}}, and change the name to
{{ZeroStripeBuffers}} and {{ZeroStripeIndices}}.
{quote}
I guess the following utilities can be moved elsewhere and shared with client
side. targetsStatus could have a better name.
{quote}
{{covertIndex4Decode}} can be shared, I will move it to {{StripedBlockUtil}}.
{quote}
I'm wondering if the following codes can be better organized, like all the
codes can be split into two functions: newStrippedReader and newBlockReader.
{quote}
The {{newBlockReader}} is already a separate function.
{quote}
Is it easy to centralize all the input/output buffers allocation in a function,
so in future it would be easier to enhance respecting the fact that Java coders
like on-heap buffer, but native coders prefer direct buffer.
{quote}
Agree, we can have a function for allocating buffer.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[
https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563090#comment-14563090
]
Kai Zheng commented on HDFS-8328:
-
Thanks Yi for working on this. The patch looks good in logic, just some
comments for better readable:
1. {{minRequiredSources}} looks like a little confusing, because from coder's
point of view, suppose 6+3, minRequiredSources would be 6 as dataBlkNum at the
first glance. Better to rename it or have some comments for it.
{code}
+ final int cellsNum = (int)((blockGroup.getNumBytes() - 1) / cellSize +
1);
+ minRequiredSources = Math.min(cellsNum, dataBlkNum);
{code}
2. How about renaming {{nullInputBuffers}} to nullCellBuffers, zeroCellBuffers
or paddingCellBuffers, as you know, in the {{inputs}} array for the decode
call, null entries indicate erasure or not to read. It could avoid some
misunderstanding. Better to have some comments for it.
{code}
+ if (minRequiredSources dataBlkNum) {
+nullInputBuffers =
+new ByteBuffer[dataBlkNum - minRequiredSources];
+nullInputIndices = new short[dataBlkNum - minRequiredSources];
+ }
{code}
3. Any better name for {{success}}? nsuccess = numSuccess or ...
{code}
+int[] success = new int[minRequiredSources];
int nsuccess = 0;
{code}
4. I guess the following utilities can be moved elsewhere and shared with
client side. {{targetsStatus}} could have a better name.
{code}
+private int[] getErasedIndices(boolean[] targetsStatus) {
+ int[] result = new int[targets.length];
+ int m = 0;
+ for (int i = 0; i targets.length; i++) {
+if (targetsStatus[i]) {
+ result[m++] = covertIndex4Decode(targetIndices[i]);
+}
+ }
+ return Arrays.copyOf(result, m);
+}
+
+private int covertIndex4Decode(int index) {
+ return index dataBlkNum ? index + parityBlkNum : index - dataBlkNum;
+}
+
{code}
5. I'm wondering if the following codes can be better organized, like all the
codes can be split into two functions: newStrippedReader and newBlockReader.
{code}
+private StripedReader addStripedReader(int i, long offset) {
+ StripedReader reader = new StripedReader(liveIndices[i]);
+ stripedReaders.add(reader);
+
+ BlockReader blockReader = newBlockReader(
+ getBlock(blockGroup, liveIndices[i]), offset, sources[i]);
+ if (blockReader != null) {
+initChecksumAndBufferSizeIfNeeded(blockReader);
+reader.blockReader = blockReader;
+ }
+ reader.buffer = ByteBuffer.allocate(bufferSize);
+ return reader;
+}
+
{code}
6. Is it easy to centralize all the input/output buffers allocation in a
function, so in future it would be easier to enhance respecting the fact that
Java coders like on-heap buffer, but native coders prefer direct buffer.
Follow-on to update decode for DataNode striped blocks reconstruction
-
Key: HDFS-8328
URL: https://issues.apache.org/jira/browse/HDFS-8328
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8328-HDFS-7285.001.patch
Current the decode for DataNode striped blocks reconstruction is a
workaround, we need to update it after the decode fix in HADOOP-11847.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555697#comment-14555697 ] Yi Liu commented on HDFS-8328: -- I will also do some refinement and improvement for blocks reconstruction in this JIRA. Follow-on to update decode for DataNode striped blocks reconstruction - Key: HDFS-8328 URL: https://issues.apache.org/jira/browse/HDFS-8328 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Current the decode for DataNode striped blocks reconstruction is a workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
