[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480908#comment-13480908 ] Hudson commented on HDFS-4022: -- Integrated in Hadoop-trunk-Commit #2904 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2904/]) HDFS-4022. Replication not happening for appended block. Contributed by Vinay. (Revision 1400578) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400578 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Vinay Priority: Blocker Fix For: 3.0.0, 2.0.3-alpha Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480948#comment-13480948 ] Hudson commented on HDFS-4022: -- Integrated in Hadoop-Yarn-trunk #10 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/10/]) HDFS-4022. Replication not happening for appended block. Contributed by Vinay. (Revision 1400578) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400578 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Vinay Priority: Blocker Fix For: 3.0.0, 2.0.3-alpha Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480960#comment-13480960 ] Hudson commented on HDFS-4022: -- Integrated in Hadoop-Hdfs-trunk #1202 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1202/]) HDFS-4022. Replication not happening for appended block. Contributed by Vinay. (Revision 1400578) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400578 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Vinay Priority: Blocker Fix For: 3.0.0, 2.0.3-alpha Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480968#comment-13480968 ] Hudson commented on HDFS-4022: -- Integrated in Hadoop-Mapreduce-trunk #1232 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1232/]) HDFS-4022. Replication not happening for appended block. Contributed by Vinay. (Revision 1400578) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400578 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Vinay Priority: Blocker Fix For: 3.0.0, 2.0.3-alpha Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480661#comment-13480661 ] Uma Maheswara Rao G commented on HDFS-4022: --- Thanks Vinay for the patch. Thanks a lot, Nicholas for your reviews. I will commit it in some time today. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Vinay Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479983#comment-13479983
]
Hadoop QA commented on HDFS-4022:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549972/HDFS-4022.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3369//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3369//console
This message is automatically generated.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch,
HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478726#comment-13478726 ] Uma Maheswara Rao G commented on HDFS-4022: --- HDFS-4072 patch incorporated to add the remove API. So, let's wait to make use of that api once HDFS-4072 in. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479614#comment-13479614 ] Suresh Srinivas commented on HDFS-4022: --- I have committed HDFS-4072. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478102#comment-13478102 ] Tsz Wo (Nicholas), SZE commented on HDFS-4022: -- replication change flow is already under lock right? You are right that it is already under lock. Thanks for pointing it out! Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477553#comment-13477553 ] Tsz Wo (Nicholas), SZE commented on HDFS-4022: -- Hi Vinay, thanks for working on this. Some comments on the patch: - PendingReplicationBlocks.remove(Block) actually only decrements numReplicasInProgress but not removes it. So I suggest: -* rename remove(Block) to decrement(Block) and -* add a new method for remove. - Need to acquire the namesystem lock before changing replication. - Let's remove the if and call pendingReplications.remove(ucBlock) anyway. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477569#comment-13477569
]
Hadoop QA commented on HDFS-4022:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1254/HDFS-4022.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3350//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3350//console
This message is automatically generated.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477594#comment-13477594
]
Uma Maheswara Rao G commented on HDFS-4022:
---
{quote}
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477593#comment-13477593
]
Uma Maheswara Rao G commented on HDFS-4022:
---
{quote}
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477606#comment-13477606
]
Uma Maheswara Rao G commented on HDFS-4022:
---
Thanks Nicholas, for the comments.
{quote}
■Need to acquire the namesystem lock before changing replication.
{quote}
replication change flow is already under lock right?
{quote}
■PendingReplicationBlocks.remove(Block)
{quote}
I agree, this is doing less than expected when NN issues multiple blocks to re
replicate. It is maintaining the index intsead of multiple pending blocks.
Let's rename it to decrement and add a straight method to delete the element.
{quote}
■Let's remove the if and call pendingReplications.remove(ucBlock) anyway.
{quote}
As things happening under the lock, I felt this is fine. It may be safe to
remove non conditionally as there is no harm. I can see a case that, when
neededReplications moved to pending replications, if one of the good block
mossed again, then NN may need to add that neededReplications.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474847#comment-13474847
]
Hadoop QA commented on HDFS-4022:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548851/HDFS-4022.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3320//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3320//console
This message is automatically generated.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474967#comment-13474967
]
Uma Maheswara Rao G commented on HDFS-4022:
---
{code}
// Start a new datanode
+ cluster.startDataNodes(conf, 1, true, null, null);
+
+ // Append to the file.
+ FSDataOutputStream append = fileSystem.append(f);
+ append.write(/testAppend.getBytes());
+ append.close();
+
{code}
you need to start DN after append and close.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch, HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474988#comment-13474988 ] Vinay commented on HDFS-4022: - Oops!! I uploaded the wrong patch.. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13475035#comment-13475035 ] Uma Maheswara Rao G commented on HDFS-4022: --- Thanks Vinay for the patch! +1 for the patch. @Nicholas, do you have any comments on the patch? I wanted to take your opinion as well before committing, as it is touching the append and replications core flows. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch, HDFS-4022.patch, HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473867#comment-13473867 ] Uma Maheswara Rao G commented on HDFS-4022: --- For the first comment, you need not do any thing for replicationIndex. I will clarify how repleIndex work. We will do that work only in Replication processing. When I pick the block for replication, we will increment that index. Once done, while removing from neededReplications after processing we may have to take care of decrementing index. So, in this case, we need not decrement it. Already code presents under namesystem lock. Need not work on first comment. Other comment is, I am seeing there would be a chance for race conditions between pendingReplications and neededReplications removal. Consider a case, when NN issued a command to DN for replication, it will move the block from neededReplication to pending replications. Before actually replicating, if user appended the block, we will try to remove it from only needeReplications as per the patch. But that will simply return fasle as block already moved to pendingReplications. I think we may have to cleanup pending replications also, otherwise after pendingReplications timeout, block may come back to neededReplications or can do similar messup. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-4022.patch Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473327#comment-13473327
]
Hadoop QA commented on HDFS-4022:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548568/HDFS-4022.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3299//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3299//console
This message is automatically generated.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[
https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473527#comment-13473527
]
Uma Maheswara Rao G commented on HDFS-4022:
---
Hi Vinay,
Comments as follows.
{code}
NumberReplicas replicas = countNodes(ucBlock);
+neededReplications.remove(ucBlock, replicas.liveReplicas(),
+replicas.decommissionedReplicas(), getReplication(ucBlock));
{code}
We may have to decrement replication index right? and keep them in
neededReplications lock.
Also you have to take care, that whether the block is really in
neededReplications list and it removed now, then only you may have to decrement
the index. Could you please check this point?
Please add timeout for test case.
Replication not happening for appended block
Key: HDFS-4022
URL: https://issues.apache.org/jira/browse/HDFS-4022
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: suja s
Assignee: Uma Maheswara Rao G
Priority: Blocker
Attachments: HDFS-4022.patch
Block written and finalized
Later append called. Block GenTS got changed.
DN side log
Can't send invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738
logged continously
NN side log
INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from
DatanodeRegistration(192.xx.xx.xx,
storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send
invalid block
BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also
logged continuosly.
The block checked for tansfer is the one with old genTS whereas the new block
with updated genTS exist in the data dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472382#comment-13472382 ] Suresh Srinivas commented on HDFS-4022: --- Suja, if you have time, can you please post a test that reproduces this problem? Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: suja s Priority: Blocker Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4022) Replication not happening for appended block
[ https://issues.apache.org/jira/browse/HDFS-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472468#comment-13472468 ] Uma Maheswara Rao G commented on HDFS-4022: --- Reproducing this case should be like, 1. File is having less replicas than expected and not yet replicated. 2. Reopen the file with append. Here genstamp will be updated to latest. And neededReplications will have the block with older genstamp still. 3. Close the file. now file is in completed state and will not update neededReplications as it already there in its list. Now replicationMonitor keep try to replicated the block with older genstamp which is already updated to newer genstamps at DN and blocksMap of NN. I think we have to remove from neededReplications as well when we move the block to InodeFileUnderConstruction. Anyway neededReplications will be updated when the block finalized at the end. tomorrow, we will get a patch for it. Replication not happening for appended block Key: HDFS-4022 URL: https://issues.apache.org/jira/browse/HDFS-4022 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: suja s Assignee: Uma Maheswara Rao G Priority: Blocker Block written and finalized Later append called. Block GenTS got changed. DN side log Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 logged continously NN side log INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Error report from DatanodeRegistration(192.xx.xx.xx, storageID=DS-2040532042-192.xx.xx.xx-50010-1348830863443, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=123456;nsid=116596173;c=0): Can't send invalid block BP-407900822-192.xx.xx.xx-1348830837061:blk_-9185630731157263852_108738 also logged continuosly. The block checked for tansfer is the one with old genTS whereas the new block with updated genTS exist in the data dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
