[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130805#comment-13130805 ] Eli Collins commented on HDFS-1779: --- Thanks Uma. I committed the fix to allow tests to run with append enabled to branch-20-security. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3-fixes.patch, > HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, HDFS-1779.2.patch, > HDFS-1779.20Append.fix.patch, HDFS-1779.2a.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112610#comment-13112610 ] Uma Maheswara Rao G commented on HDFS-1779: --- Thanks a lot Eli for updating the patch. Even though we are running the tests with append off, better to handle it. In future if we enable append by default, again we need to fix this Test Code. Good point, lets handle now itself. I have updated the patch for 20Append as well. Patch looks good to me. +1 from my side. Thanks Uma > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3-fixes.patch, > HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, HDFS-1779.2.patch, > HDFS-1779.20Append.fix.patch, HDFS-1779.2a.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112268#comment-13112268 ] Eli Collins commented on HDFS-1779: --- Also, SimulatedFSDataset#getBlocksBeingWrittenReport should return new Block[0] not null, otherwise Datanode#register will pass null to BlockListAsLongs#convertToArrayLongs causing an NPE. You probably didn't see this on branch 20 security because it's not running with append support defaulted to true. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108857#comment-13108857 ] Uma Maheswara Rao G commented on HDFS-1779: --- Thanks Eli, I will take a look on this. Thanks Uma > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108827#comment-13108827 ] Eli Collins commented on HDFS-1779: --- Nit, in recoverBlocksBeingWritten we should remove the comment wrt sending block received messages since this was removed. - * ongoingCreates. Also, send a blockreceived message to the NN - * for each of these blocks because these are not part of a - * block report. + * ongoingCreates. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104125#comment-13104125 ] Uma Maheswara Rao G commented on HDFS-1779: --- Hi Todd/Jitendra, Can you please take a look for 20Append? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103817#comment-13103817 ] Uma Maheswara Rao G commented on HDFS-1779: --- Can some one commit on 20Append also? Thanks Uma > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101505#comment-13101505 ] Jitendra Nath Pandey commented on HDFS-1779: Commited to branch-0.20-security. Thanks to Uma! > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101496#comment-13101496 ] Suresh Srinivas commented on HDFS-1779: --- +1 for the patch. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101475#comment-13101475 ] Uma Maheswara Rao G commented on HDFS-1779: --- Hi Jitendra, Thanks for the Review and security Patch! I just gone through the patch! +1 from my side. It looks good. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101469#comment-13101469 ] Hadoop QA commented on HDFS-1779: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493838/HDFS-1779-20.security.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1226//console This message is automatically generated. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779-20.security.3.patch, HDFS-1779.1.patch, > HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101005#comment-13101005 ] Jitendra Nath Pandey commented on HDFS-1779: +1 for the patch. This needs to be ported to 20-security branch as well. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, > HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099184#comment-13099184 ] Hadoop QA commented on HDFS-1779: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493354/HDFS-1779.2a.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1219//console This message is automatically generated. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, > HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098812#comment-13098812 ] Jitendra Nath Pandey commented on HDFS-1779: A few comments: TestAppend.java: # I will recommend to rename the class to something like TestBBWBlockReport. # Please use cluster.restartNameNode() instead of private restartNamenode. # Can we avoid making dfs, namenode, outStream etc as class fields. # In tearDown, namenode.stop is not required because cluster.shutdown takes care of it. # outStream.close will likely fail after testDNShouldNotSendBBWReportIfAppendOff because namenode will be in safemode. # waitFor method can be replaced with Thread.sleep. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098486#comment-13098486 ] Hairong Kuang commented on HDFS-1779: - With this patch, blockReceived does not get sent for bbw blocks. But my patch does not clean up the code for blockReceived. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098461#comment-13098461 ] Sanjay Radia commented on HDFS-1779: Since this patch sends a separate BBW-report on each DN registration, one should not need the "block received" that are being sent by DN introduced as part of hdfs-142 patch. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098350#comment-13098350 ] Hairong Kuang commented on HDFS-1779: - Uma, thanks for addressing Todd's review comments and added unit tests. Todd, could you please review this one more time? Thanks! > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097216#comment-13097216 ] Hadoop QA commented on HDFS-1779: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493045/HDFS-1779.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1195//console This message is automatically generated. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097214#comment-13097214 ] Uma Maheswara Rao G commented on HDFS-1779: --- Hi Todd & Hairong, Added the additional tests to reproduce this scenario! Patch contains: Hairong code changes + Todd review comments fixes + Added additional testcases for reproducing this scenario. + made the supportAppends variable to private in datanode and added javadoc for processBlocksBeingWritten api. One more scenario is, if more number of blocks are under construction and also sync has been called for blocks .After that NN restart happened. Then if DNs didn't send the BBW report to NN, it will never become out of safemode because NN will expect some threshold of blocks to report to come out of safemode.This condition also added in tests. Thanks Uma > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append, 0.20.205.0 > > Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.patch, > bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096449#comment-13096449 ] Todd Lipcon commented on HDFS-1779: --- Mostly looks great. Small nits: - indentation in SimulatedFSDataset - there are some hard tabs in rejectAddStoredBlock - typo: 'unregisterted' Hairong, do you have a unit test? I have half of one here, similar to Uma's. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096163#comment-13096163 ] Hadoop QA commented on HDFS-1779: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492778/bbwReportAppend.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1188//console This message is automatically generated. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch, bbwReportAppend.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093994#comment-13093994 ] Uma Maheswara Rao G commented on HDFS-1779: --- Hi Hairong, Actually i was also started looking into this. Since you have patch ready, I will test your patch. Thanks a lot Hairong for the help :-) Thanks Uma > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093943#comment-13093943 ] Hairong Kuang commented on HDFS-1779: - Sure, I can upload a patch for supporting bbw block report. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[
https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092653#comment-13092653
]
Uma Maheswara Rao G commented on HDFS-1779:
---
Thanks a lot to Todd and Hairong for review!
{quote}
since that puts it into the blockmap but not the "targets" array. Wouldn't it
be more correct to put it into the INodeFileUnderConstruction's targets array?
{quote}
Good point. I will take a look.
I will update the patch as per your comments. or Hairong, do you have patch
ready?
Thanks,
Uma
> After NameNode restart , Clients can not read partial files even after client
> invokes Sync.
> ---
>
> Key: HDFS-1779
> URL: https://issues.apache.org/jira/browse/HDFS-1779
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
>Affects Versions: 0.20-append
> Environment: Linux
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.20-append
>
> Attachments: HDFS-1779.1.patch, HDFS-1779.patch
>
>
> In Append HDFS-200 issue,
> If file has 10 blocks and after writing 5 blocks if client invokes sync
> method then NN will persist the blocks information in edits.
> After this if we restart the NN, All the DataNodes will reregister with NN.
> But DataNodes are not sending the blocks being written information to NN. DNs
> are sending the blocksBeingWritten information in DN startup. So, here
> NameNode can not find that the 5 persisted blocks belongs to which datanodes.
> This information can build based on block reports from DN. Otherwise we will
> loose this 5 blocks information even NN persisted that block information in
> edits.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092647#comment-13092647 ] Hairong Kuang commented on HDFS-1779: - Yes, when processing a bbw block report, we put bbw replicas into tagets arrays but not in blocksMap. Basically with a separate bbw block report, addStoredBlock could be made much cleaner in append 0.20 and a bbw replica can be handled more semantic correctly. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092645#comment-13092645 ] Todd Lipcon commented on HDFS-1779: --- Hi Uma, Hairong. I've been looking at this a bit here as well. I think it's not quite right to use blockReceived here - since that puts it into the blockmap but not the "targets" array. Wouldn't it be more correct to put it into the INodeFileUnderConstruction's targets array? Hairong: is that what your change does? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092639#comment-13092639 ] Uma Maheswara Rao G commented on HDFS-1779: --- why do we need separate report for bbw? I think recoverBlcoksBeingWritten call will be enough upon registration.. no? Can you please just review the patch? If any suggestion from you, i will update the patch accordingly. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092633#comment-13092633 ] Uma Maheswara Rao G commented on HDFS-1779: --- Hi Hairong, I did not get what change your are suggesting here. Can you please give hint, what change you are expecting in this patch? Anyway i need to rebase it on trunk. If suggest the change i will be happy to do that :-) Thanks Uma > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092632#comment-13092632 ] Hairong Kuang commented on HDFS-1779: - In our fb internal branch, we fixed the problem by sending a separate bbw report upon registration. Our solution is cleaner but this patch requires less code change. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082602#comment-13082602 ] Hadoop QA commented on HDFS-1779: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12489473/HDFS-1779.1.patch against trunk revision 1155998. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1070//console This message is automatically generated. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.1.patch, HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037314#comment-13037314 ] Uma Maheswara Rao G commented on HDFS-1779: --- Yes Dhruba. The patch does the same thing that you have mentioned. DN sends the blocks being written to the NN on restart scenario only. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037012#comment-13037012 ] dhruba borthakur commented on HDFS-1779: if the namenode restarts, the datanode should invoke FSDataSet.recoverBlocksBeingWritten(). This will send all blocks in the block being written directory to the NN. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036901#comment-13036901 ] Uma Maheswara Rao G commented on HDFS-1779: --- This patch is for 0.20-Append branch > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036897#comment-13036897 ] Hadoop QA commented on HDFS-1779: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479917/HDFS-1779.patch against trunk revision 1125217. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/602//console This message is automatically generated. > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.20-append > > Attachments: HDFS-1779.patch > > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014041#comment-13014041 ] Uma Maheswara Rao G commented on HDFS-1779: --- Thanks to Druba and Todd for spending time and giving comments. Datanodes already sends their blockRecieved command to NN successfull blocks. After this, NN restart happend then this problem problem can come. Because DataNode will not send any information about bbw.So NN can not find this partial blcok presents in which node. My proposal would be , When namenode is restarted, the datanode will re-register with the namenode. During datanode registration, can we recover the blocks in blocksBeingWritten directory ( sending bbw details to NN)? I verified this with two DNs, it works for normal scenarios. Do you think any impact with this ? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011584#comment-13011584 ] dhruba borthakur commented on HDFS-1779: If the app has invoked sync/hflush (as specified in the jira description), then the block *is* persisted in the fsimage. Lease recovery won't be successful until the time the datanode sends those blocks as part of a block report, So this is a bug and the workaround to get the missing blocks back would be to restart the datanodes? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011569#comment-13011569 ] Todd Lipcon commented on HDFS-1779: --- Dhruba -- since we don't serialize the targets[] array with the lease, I don't think the NN could actually trigger block recovery properly. Am I missing something? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011567#comment-13011567 ] dhruba borthakur commented on HDFS-1779: Wait a minute, can you pl provide some more clarification? If you restart only the NN and not the DN, then the DN still has a reference to the blocks in the bbw directory. When the client closes the file (assuming that the client survived the NN restart), those blocks will send a blockReceived to the NN. If the client died, then the NN will start lease-recovery at some future point in time and that should make these datanodes send blockReceived messages to the NN. This sequence of events should make the blocks reappear in the NN. Are you seeing something different? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011565#comment-13011565 ] dhruba borthakur commented on HDFS-1779: This seems to be a valid bug, would you have a patch for it? > After NameNode restart , Clients can not read partial files even after client > invokes Sync. > --- > > Key: HDFS-1779 > URL: https://issues.apache.org/jira/browse/HDFS-1779 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Affects Versions: 0.20-append > Environment: Linux >Reporter: Uma Maheswara Rao G > > In Append HDFS-200 issue, > If file has 10 blocks and after writing 5 blocks if client invokes sync > method then NN will persist the blocks information in edits. > After this if we restart the NN, All the DataNodes will reregister with NN. > But DataNodes are not sending the blocks being written information to NN. DNs > are sending the blocksBeingWritten information in DN startup. So, here > NameNode can not find that the 5 persisted blocks belongs to which datanodes. > This information can build based on block reports from DN. Otherwise we will > loose this 5 blocks information even NN persisted that block information in > edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
