[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss
[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873238#action_12873238 ] dhruba borthakur commented on HDFS-1094: >From all the above mathematical formulae, can we then say that the policy >listed below can be a good first step: "The first replica is on a node on rack r. Then the other two replicas be randomly selected nodes on either rack r-1 or r+1. In this approach, three datanodes in two consecutive racks have to fail simultaneously for a block loss to occur. This is better than the current implementation where any three datanode failures in the entire cluster can cause one block to be lost." The NetworkTopology class in the namenode already has information about node to rack location. > Intelligent block placement policy to decrease probability of block loss > > > Key: HDFS-1094 > URL: https://issues.apache.org/jira/browse/HDFS-1094 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: Rodrigo Schmidt > Attachments: prob.pdf, prob.pdf > > > The current HDFS implementation specifies that the first replica is local and > the other two replicas are on any two random nodes on a random remote rack. > This means that if any three datanodes die together, then there is a > non-trivial probability of losing at least one block in the cluster. This > JIRA is to discuss if there is a better algorithm that can lower probability > of losing a block. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1057) Concurrent readers hit ChecksumExceptions if following a writer to very end of file
[ https://issues.apache.org/jira/browse/HDFS-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam rash updated HDFS-1057: --- Attachment: hdfs-1057-trunk-1.txt ported patch to trunk (hairong's idea of storing last checksum) > Concurrent readers hit ChecksumExceptions if following a writer to very end > of file > --- > > Key: HDFS-1057 > URL: https://issues.apache.org/jira/browse/HDFS-1057 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: sam rash >Priority: Blocker > Attachments: conurrent-reader-patch-1.txt, > conurrent-reader-patch-2.txt, conurrent-reader-patch-3.txt, > hdfs-1057-trunk-1.txt > > > In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before > calling flush(). Therefore, if there is a concurrent reader, it's possible to > race here - the reader will see the new length while those bytes are still in > the buffers of BlockReceiver. Thus the client will potentially see checksum > errors or EOFs. Additionally, the last checksum chunk of the file is made > accessible to readers even though it is not stable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1027) Update year to 2010.
[
https://issues.apache.org/jira/browse/HDFS-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HDFS-1027:
--
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: (was: 0.21.0)
(was: 0.20.1)
(was: 0.20.2)
(was: 0.20.3)
Resolution: Fixed
+1. I've committed this. Thanks, Ravi. Resolving as fixed.
> Update year to 2010.
> -
>
> Key: HDFS-1027
> URL: https://issues.apache.org/jira/browse/HDFS-1027
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.20.1, 0.20.2
>Reporter: Ravi Phulari
>Assignee: Ravi Phulari
>Priority: Trivial
> Fix For: 0.22.0
>
> Attachments: HDFS-1027.patch
>
>
> Copyright year needs to be updated from 2009 to 2010.
> {code:xml}
>
> 2009
> The Apache Software Foundation.
> http://www.apache.org/licenses/
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873212#action_12873212 ] sam rash commented on HDFS-1142: konstantin : nvm last comment, i misread your comment (and forgot 22 == trunk) > Lease recovery doesn't reassign lease when triggered by append() > > > Key: HDFS-1142 > URL: https://issues.apache.org/jira/browse/HDFS-1142 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-1142.txt, hdfs-1142.txt > > > If a soft lease has expired and another writer calls append(), it triggers > lease recovery but doesn't reassign the lease to a new owner. Therefore, the > old writer can continue to allocate new blocks, try to steal back the lease, > etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss
[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt reassigned HDFS-1094: - Assignee: Rodrigo Schmidt (was: dhruba borthakur) > Intelligent block placement policy to decrease probability of block loss > > > Key: HDFS-1094 > URL: https://issues.apache.org/jira/browse/HDFS-1094 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: Rodrigo Schmidt > Attachments: prob.pdf, prob.pdf > > > The current HDFS implementation specifies that the first replica is local and > the other two replicas are on any two random nodes on a random remote rack. > This means that if any three datanodes die together, then there is a > non-trivial probability of losing at least one block in the cluster. This > JIRA is to discuss if there is a better algorithm that can lower probability > of losing a block. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1177) Ivy2.0 has bugs: let's upgrate to 2.1.0
[ https://issues.apache.org/jira/browse/HDFS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1177: -- Hadoop Flags: [Reviewed] +1, although we may as well delete that previous, commented out version of ivy as well. Common's already been moved to this version; we should open a corresponding JIRA for MR. It's best to have all the kids using the same toys. > Ivy2.0 has bugs: let's upgrate to 2.1.0 > --- > > Key: HDFS-1177 > URL: https://issues.apache.org/jira/browse/HDFS-1177 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HADOOP-6792.patch > > > Ivy before 2.1 has bugs in checksums calculation from sha1 files. It might > prevent the build from getting some artifacts. Let's upgrade to Ivy 2.1.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1134) Large-scale Automated Framework
[ https://issues.apache.org/jira/browse/HDFS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873165#action_12873165 ] Hadoop QA commented on HDFS-1134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445801/HDFS-1134.patch against trunk revision 949084. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 37 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/385/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/385/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/385/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/385/console This message is automatically generated. > Large-scale Automated Framework > --- > > Key: HDFS-1134 > URL: https://issues.apache.org/jira/browse/HDFS-1134 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch > > > This is HDFS part of HADOOP-6332 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1178) The NameNode servlets should not use RPC to connect to the NameNode
[ https://issues.apache.org/jira/browse/HDFS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873163#action_12873163 ] Jakob Homan commented on HDFS-1178: --- +1 for direct-nn.patch. > The NameNode servlets should not use RPC to connect to the NameNode > --- > > Key: HDFS-1178 > URL: https://issues.apache.org/jira/browse/HDFS-1178 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: direct-nn.patch, hdfs-1178-y20.patch, hdfs-1178-y20.patch > > > Currently some of the NameNode servlets use RPC to connect from the NameNode > to itself. They should do it more directly with the NameNode object. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1178) The NameNode servlets should not use RPC to connect to the NameNode
[ https://issues.apache.org/jira/browse/HDFS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-1178: Attachment: direct-nn.patch I fixed the error handling. This is for y20, so don't commit it. > The NameNode servlets should not use RPC to connect to the NameNode > --- > > Key: HDFS-1178 > URL: https://issues.apache.org/jira/browse/HDFS-1178 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: direct-nn.patch, hdfs-1178-y20.patch, hdfs-1178-y20.patch > > > Currently some of the NameNode servlets use RPC to connect from the NameNode > to itself. They should do it more directly with the NameNode object. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1134) Large-scale Automated Framework
[ https://issues.apache.org/jira/browse/HDFS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1134: - Status: Patch Available (was: Open) > Large-scale Automated Framework > --- > > Key: HDFS-1134 > URL: https://issues.apache.org/jira/browse/HDFS-1134 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch > > > This is HDFS part of HADOOP-6332 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1134) Large-scale Automated Framework
[
https://issues.apache.org/jira/browse/HDFS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-1134:
-
Attachment: HDFS-1134.patch
{{test-system}} is running against a real cluster. To avoid the hassle of
configuring a new 0.22 I have simply used an instrumented 0.20 cluster. Since
the code is the same for both the test is valid. And it has passed normally.
> Large-scale Automated Framework
> ---
>
> Key: HDFS-1134
> URL: https://issues.apache.org/jira/browse/HDFS-1134
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.21.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Attachments: HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch,
> HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch,
> HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch,
> HDFS-1134.patch
>
>
> This is HDFS part of HADOOP-6332
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1134) Large-scale Automated Framework
[ https://issues.apache.org/jira/browse/HDFS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1134: - Status: Open (was: Patch Available) > Large-scale Automated Framework > --- > > Key: HDFS-1134 > URL: https://issues.apache.org/jira/browse/HDFS-1134 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch > > > This is HDFS part of HADOOP-6332 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1179) Implement a file change log
[ https://issues.apache.org/jira/browse/HDFS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873089#action_12873089 ] Allen Wittenauer commented on HDFS-1179: Agreed: this sounds exactly like the audit log facility. But it is important to have it exist outside the HDFS as well (debugging during a crash, security, etc), so we need to be able to do *both*. > Implement a file change log > --- > > Key: HDFS-1179 > URL: https://issues.apache.org/jira/browse/HDFS-1179 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > The File Change Log (FCL) is a construct that records all metadata changes in > a file system (http://en.wikipedia.org/wiki/File_change_log). This JIRA is to > discuss various high-level approaches to implementing such a FCL for HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1179) Implement a file change log
[ https://issues.apache.org/jira/browse/HDFS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873081#action_12873081 ] Konstantin Shvachko commented on HDFS-1179: --- This sounds like auditLog. Is there a difference? It would be nice to put auditLogs into hdfs. > Implement a file change log > --- > > Key: HDFS-1179 > URL: https://issues.apache.org/jira/browse/HDFS-1179 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > The File Change Log (FCL) is a construct that records all metadata changes in > a file system (http://en.wikipedia.org/wiki/File_change_log). This JIRA is to > discuss various high-level approaches to implementing such a FCL for HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1039) Service should be set in the token in JspHelper.getUGI
[ https://issues.apache.org/jira/browse/HDFS-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873074#action_12873074 ] Jitendra Nath Pandey commented on HDFS-1039: ant test was run manually. All tests pass except TestHDFSTrash which also fails without this patch. > Service should be set in the token in JspHelper.getUGI > -- > > Key: HDFS-1039 > URL: https://issues.apache.org/jira/browse/HDFS-1039 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1039-y20.1.patch, HDFS-1039-y20.2.1.patch, > HDFS-1039-y20.2.patch, HDFS-1039.2.patch, HDFS-1039.4.patch > > > The delegation token added to the UGI in getUGI method in the JspHelper does > not have service set. Therefore, this token cannot be used to connect to the > namenode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1179) Implement a file change log
[ https://issues.apache.org/jira/browse/HDFS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873066#action_12873066 ] Dmytro Molkov commented on HDFS-1179: - Philip, potentially it can be implemented this way, however it depends on the requirements from the end consumer of this change log. In case of RAID it will work perfectly fine with the change log coming at the time of edits roll. However if there are processes that will benefit from this change and they need more real time data it seems like this should be implemented in the NameNode itself. > Implement a file change log > --- > > Key: HDFS-1179 > URL: https://issues.apache.org/jira/browse/HDFS-1179 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > The File Change Log (FCL) is a construct that records all metadata changes in > a file system (http://en.wikipedia.org/wiki/File_change_log). This JIRA is to > discuss various high-level approaches to implementing such a FCL for HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1179) Implement a file change log
[ https://issues.apache.org/jira/browse/HDFS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873046#action_12873046 ] Philip Zeyliger commented on HDFS-1179: --- Conceptually, this could be a side output of whoever rolls up the image+edits files into a new image, yes? -- Philip > Implement a file change log > --- > > Key: HDFS-1179 > URL: https://issues.apache.org/jira/browse/HDFS-1179 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > The File Change Log (FCL) is a construct that records all metadata changes in > a file system (http://en.wikipedia.org/wiki/File_change_log). This JIRA is to > discuss various high-level approaches to implementing such a FCL for HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1134) Large-scale Automated Framework
[ https://issues.apache.org/jira/browse/HDFS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872926#action_12872926 ] Sharad Agarwal commented on HDFS-1134: -- Tested targets -> jar-system, jar-test-system. Worked fine. However test-system failed with this: src/test/aop/build/aop.xml:138: required attribute suite.type not set Could be due to some parameter not being passed. Please manually verify that test-system passes against a real cluster. Barring this, patch looks fine to me. > Large-scale Automated Framework > --- > > Key: HDFS-1134 > URL: https://issues.apache.org/jira/browse/HDFS-1134 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch > > > This is HDFS part of HADOOP-6332 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1134) Large-scale Automated Framework
[ https://issues.apache.org/jira/browse/HDFS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872900#action_12872900 ] Hadoop QA commented on HDFS-1134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445748/HDFS-1134.patch against trunk revision 949084. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 48 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/384/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/384/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/384/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/384/console This message is automatically generated. > Large-scale Automated Framework > --- > > Key: HDFS-1134 > URL: https://issues.apache.org/jira/browse/HDFS-1134 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, > HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch, HDFS-1134.patch > > > This is HDFS part of HADOOP-6332 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
