[jira] [Commented] (HDFS-4712) New libhdfs method hdfsGetDataNodes
[ https://issues.apache.org/jira/browse/HDFS-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644406#comment-13644406 ] andrea manzi commented on HDFS-4712: Hi Colin, thanks a lot for your comments and for the tips they were really useful and now my code looks better. ( i hope:-)) 2 things: 1) i have kept the signature with numEntries in the hdfsFreeDataNodeInfo method, cause i aligned the method to hdfsFreeFileInfo. But i can also remove the parameter and use a null pointer as you suggested 2) i added a method (enum_to_string) to convert the enum value to a string, cause in the generation of the enumtype trough JNI to pass as parameter to getDataNodeStats, i need the string associated to the enum value. i have also created a patch as described in the wiki and attached to the ticket thanks again Andrea > New libhdfs method hdfsGetDataNodes > --- > > Key: HDFS-4712 > URL: https://issues.apache.org/jira/browse/HDFS-4712 > Project: Hadoop HDFS > Issue Type: New Feature > Components: libhdfs >Reporter: andrea manzi > Attachments: HDFS-4712.patch, hdfs.c.diff, hdfs.h.diff > > > we have implemented a possible extension to libhdfs to retrieve information > about the available datanodes ( there was a mail on the hadoop-hdsf-dev > mailing list initially abut this : > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201204.mbox/%3CCANhO- > s0mvororrxpjnjbql6brkj4c7l+u816xkdc+2r0whj...@mail.gmail.com%3E) > i would like to know how to proceed to create a patch, cause on the wiki > http://wiki.apache.org/hadoop/HowToContribute i can see info about JAVA > patches but nothing related to extensions in C. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4712) New libhdfs method hdfsGetDataNodes
[ https://issues.apache.org/jira/browse/HDFS-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrea manzi updated HDFS-4712: --- Attachment: HDFS-4712.patch > New libhdfs method hdfsGetDataNodes > --- > > Key: HDFS-4712 > URL: https://issues.apache.org/jira/browse/HDFS-4712 > Project: Hadoop HDFS > Issue Type: New Feature > Components: libhdfs >Reporter: andrea manzi > Attachments: HDFS-4712.patch, hdfs.c.diff, hdfs.h.diff > > > we have implemented a possible extension to libhdfs to retrieve information > about the available datanodes ( there was a mail on the hadoop-hdsf-dev > mailing list initially abut this : > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201204.mbox/%3CCANhO- > s0mvororrxpjnjbql6brkj4c7l+u816xkdc+2r0whj...@mail.gmail.com%3E) > i would like to know how to proceed to create a patch, cause on the wiki > http://wiki.apache.org/hadoop/HowToContribute i can see info about JAVA > patches but nothing related to extensions in C. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4734) HDFS Tests that use ShellCommandFencer are broken on Windows
[ https://issues.apache.org/jira/browse/HDFS-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644410#comment-13644410 ] Hudson commented on HDFS-4734: -- Integrated in Hadoop-Yarn-trunk #198 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/198/]) HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows. Contributed by Arpit Agarwal. (Revision 1476877) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476877 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java > HDFS Tests that use ShellCommandFencer are broken on Windows > > > Key: HDFS-4734 > URL: https://issues.apache.org/jira/browse/HDFS-4734 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 3.0.0 > > Attachments: HDFS-4734.001.patch, HDFS-4734.002.patch, > HDFS-4734.003.patch > > > The following tests use the POSIX true/false commands which are not available > on Windows. > # TestDFSHAAdmin > # TestDFSHAAdminMiniCluster > # TestNodeFencer > Additionally, ShellCommandFencer has a hard-coded dependency on bash (also > documented at > https://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4734) HDFS Tests that use ShellCommandFencer are broken on Windows
[ https://issues.apache.org/jira/browse/HDFS-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644464#comment-13644464 ] Hudson commented on HDFS-4734: -- Integrated in Hadoop-Hdfs-trunk #1387 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1387/]) HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows. Contributed by Arpit Agarwal. (Revision 1476877) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476877 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java > HDFS Tests that use ShellCommandFencer are broken on Windows > > > Key: HDFS-4734 > URL: https://issues.apache.org/jira/browse/HDFS-4734 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 3.0.0 > > Attachments: HDFS-4734.001.patch, HDFS-4734.002.patch, > HDFS-4734.003.patch > > > The following tests use the POSIX true/false commands which are not available > on Windows. > # TestDFSHAAdmin > # TestDFSHAAdminMiniCluster > # TestNodeFencer > Additionally, ShellCommandFencer has a hard-coded dependency on bash (also > documented at > https://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4734) HDFS Tests that use ShellCommandFencer are broken on Windows
[ https://issues.apache.org/jira/browse/HDFS-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644500#comment-13644500 ] Hudson commented on HDFS-4734: -- Integrated in Hadoop-Mapreduce-trunk #1414 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1414/]) HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows. Contributed by Arpit Agarwal. (Revision 1476877) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476877 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java > HDFS Tests that use ShellCommandFencer are broken on Windows > > > Key: HDFS-4734 > URL: https://issues.apache.org/jira/browse/HDFS-4734 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 3.0.0 > > Attachments: HDFS-4734.001.patch, HDFS-4734.002.patch, > HDFS-4734.003.patch > > > The following tests use the POSIX true/false commands which are not available > on Windows. > # TestDFSHAAdmin > # TestDFSHAAdminMiniCluster > # TestNodeFencer > Additionally, ShellCommandFencer has a hard-coded dependency on bash (also > documented at > https://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4734) HDFS Tests that use ShellCommandFencer are broken on Windows
[ https://issues.apache.org/jira/browse/HDFS-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644573#comment-13644573 ] Arpit Agarwal commented on HDFS-4734: - Thanks Suresh and Chris for reviewing! > HDFS Tests that use ShellCommandFencer are broken on Windows > > > Key: HDFS-4734 > URL: https://issues.apache.org/jira/browse/HDFS-4734 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 3.0.0 > > Attachments: HDFS-4734.001.patch, HDFS-4734.002.patch, > HDFS-4734.003.patch > > > The following tests use the POSIX true/false commands which are not available > on Windows. > # TestDFSHAAdmin > # TestDFSHAAdminMiniCluster > # TestNodeFencer > Additionally, ShellCommandFencer has a hard-coded dependency on bash (also > documented at > https://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644593#comment-13644593 ] Chris Nauroth commented on HDFS-4610: - This patch can't go to branch-2 yet. It's dependent on HADOOP-9413, which is dependent on hadoop.dll for native I/O on Windows. That code doesn't exist in branch-2. > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4578) Restrict snapshot IDs to 24-bits wide
[ https://issues.apache.org/jira/browse/HDFS-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4578: - Priority: Minor (was: Major) Hadoop Flags: Reviewed +1 patch looks good. > Restrict snapshot IDs to 24-bits wide > - > > Key: HDFS-4578 > URL: https://issues.apache.org/jira/browse/HDFS-4578 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Minor > Fix For: Snapshot (HDFS-2802) > > Attachments: HDFS-4578.004.patch, HDFS-4578.patch, HDFS-4578.patch, > HDFS-4578.patch > > > Snapshot IDs will be restricted to 24-bits. This will allow at the most > ~16Million snapshots globally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4578) Restrict snapshot IDs to 24-bits wide
[ https://issues.apache.org/jira/browse/HDFS-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4578. -- Resolution: Fixed I have committed this. Thanks, Arpit! > Restrict snapshot IDs to 24-bits wide > - > > Key: HDFS-4578 > URL: https://issues.apache.org/jira/browse/HDFS-4578 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Minor > Fix For: Snapshot (HDFS-2802) > > Attachments: HDFS-4578.004.patch, HDFS-4578.patch, HDFS-4578.patch, > HDFS-4578.patch > > > Snapshot IDs will be restricted to 24-bits. This will allow at the most > ~16Million snapshots globally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4773) Fix bugs in quota usage updating/computation
[ https://issues.apache.org/jira/browse/HDFS-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644652#comment-13644652 ] Tsz Wo (Nicholas), SZE commented on HDFS-4773: -- - getDiffById(final int snapshotId) and getDiff(Snapshot snapshot) are very similar. Could you change getDiff to call getDiffById? - remove the commented code in FileWithSnapshot. - Could you also fix TestOfflineImageViewer? > Fix bugs in quota usage updating/computation > > > Key: HDFS-4773 > URL: https://issues.apache.org/jira/browse/HDFS-4773 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-4773.001.patch > > > 1. FileWithSnapshot#updateQuotaAndCollectBlocks did not consider the scenario > that all the snapshots has been deleted from a snapshot copy of deleted file. > This may lead to a divide-by-0 error. > 2. When computing the quota usage for a WithName node and its subtree, if the > snapshot associated with the WithName node at the time of rename operation > has been deleted, we should compute the quota based on the posterior snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4578) Restrict snapshot IDs to 24-bits wide
[ https://issues.apache.org/jira/browse/HDFS-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644682#comment-13644682 ] Arpit Agarwal commented on HDFS-4578: - Thanks Nicholas! > Restrict snapshot IDs to 24-bits wide > - > > Key: HDFS-4578 > URL: https://issues.apache.org/jira/browse/HDFS-4578 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Minor > Fix For: Snapshot (HDFS-2802) > > Attachments: HDFS-4578.004.patch, HDFS-4578.patch, HDFS-4578.patch, > HDFS-4578.patch > > > Snapshot IDs will be restricted to 24-bits. This will allow at the most > ~16Million snapshots globally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4774) Backport HDFS-4525 'Provide an API for knowing that whether file is closed or not' to branch 1.1
Ted Yu created HDFS-4774: Summary: Backport HDFS-4525 'Provide an API for knowing that whether file is closed or not' to branch 1.1 Key: HDFS-4774 URL: https://issues.apache.org/jira/browse/HDFS-4774 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu HDFS-4525 compliments lease recovery API which allows user to know whether the recovery has completed. This JIRA backports the API to branch 1.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4774) Backport HDFS-4525 'Provide an API for knowing that whether file is closed or not' to branch 1.1
[ https://issues.apache.org/jira/browse/HDFS-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HDFS-4774: - Attachment: 4774.txt > Backport HDFS-4525 'Provide an API for knowing that whether file is closed or > not' to branch 1.1 > > > Key: HDFS-4774 > URL: https://issues.apache.org/jira/browse/HDFS-4774 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 4774.txt > > > HDFS-4525 compliments lease recovery API which allows user to know whether > the recovery has completed. > This JIRA backports the API to branch 1.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4733) Make HttpFS username pattern configurable
[ https://issues.apache.org/jira/browse/HDFS-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644693#comment-13644693 ] Aaron T. Myers commented on HDFS-4733: -- Thanks, Daryn. I agree with what you're saying in principle, but that seems like a much more involved change for this little issue. Not sure it's worth it. +1, I'm going to commit this momentarily. > Make HttpFS username pattern configurable > - > > Key: HDFS-4733 > URL: https://issues.apache.org/jira/browse/HDFS-4733 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-4733.patch, HDFS-4733.patch, HDFS-4733.patch > > > Continuing with the saga, valid usernames seem to be quite different across > unix distributions. Now running into the case of a setup where they use full > numeric names. > We should make the username pattern configurable via httpfs-site.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4712) New libhdfs method hdfsGetDataNodes
[
https://issues.apache.org/jira/browse/HDFS-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644694#comment-13644694
]
Colin Patrick McCabe commented on HDFS-4712:
bq. +dataNodeInfoList = calloc(jNumDataNodeInfos,
sizeof(hdfsDataNodeInfo**));
Should be sizeof(hdfsDataNodeInfo*)... although it doesn't matter, since those
pointers will be the same size :)
bq.+dataNodeInfos->location =
+ (const char*)((*env)->GetStringUTFChars(env, jVal.l, NULL));
Need to check to make sure this didn't fail. If it failed, an exception will
be raised. I suggest using {{newCStr}}, which handles the exception issues for
you.
bq. +dataNodeInfoList[i] = (hdfsDataNodeInfo*)
malloc(sizeof(hdfsDataNodeInfo));
You need to check if this failed (we check all the other allocation sites.)
{code}
+if (jExc) {
+printExceptionAndFree( env,jExc,PRINT_EXC_ALL,
+"org.apache.hadoop.hdfs.DistributedFileSystem::getDataNodeStats");
+errno = EPERM;
+goto done;
+}
+
+
+jobjectArray jDatanodeInfos = NULL;
{code}
Please don't use C99-style declarations after the start of the function. They
don't work well with gotos. For example, in this case, if you follow that
goto, jDatanodeInfos has undefined contents (which is likely to cause crashes
when you try to clean up). Put all the declarations at the top, except scoped
declarations.
> New libhdfs method hdfsGetDataNodes
> ---
>
> Key: HDFS-4712
> URL: https://issues.apache.org/jira/browse/HDFS-4712
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: libhdfs
>Reporter: andrea manzi
> Attachments: HDFS-4712.patch, hdfs.c.diff, hdfs.h.diff
>
>
> we have implemented a possible extension to libhdfs to retrieve information
> about the available datanodes ( there was a mail on the hadoop-hdsf-dev
> mailing list initially abut this :
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201204.mbox/%3CCANhO-
> s0mvororrxpjnjbql6brkj4c7l+u816xkdc+2r0whj...@mail.gmail.com%3E)
> i would like to know how to proceed to create a patch, cause on the wiki
> http://wiki.apache.org/hadoop/HowToContribute i can see info about JAVA
> patches but nothing related to extensions in C.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4733) Make HttpFS username pattern configurable
[ https://issues.apache.org/jira/browse/HDFS-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4733: - Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Tucu, and thanks also to Chris and Daryn for the reviews. > Make HttpFS username pattern configurable > - > > Key: HDFS-4733 > URL: https://issues.apache.org/jira/browse/HDFS-4733 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.5-beta > > Attachments: HDFS-4733.patch, HDFS-4733.patch, HDFS-4733.patch > > > Continuing with the saga, valid usernames seem to be quite different across > unix distributions. Now running into the case of a setup where they use full > numeric names. > We should make the username pattern configurable via httpfs-site.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4733) Make HttpFS username pattern configurable
[ https://issues.apache.org/jira/browse/HDFS-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644706#comment-13644706 ] Hudson commented on HDFS-4733: -- Integrated in Hadoop-trunk-Commit #3689 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3689/]) HDFS-4733. Make HttpFS username pattern configurable. Contributed by Alejandro Abdelnur. (Revision 1477237) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477237 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebApp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/UserProvider.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSCustomUserName.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/lib/wsrs/TestUserProvider.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Make HttpFS username pattern configurable > - > > Key: HDFS-4733 > URL: https://issues.apache.org/jira/browse/HDFS-4733 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.5-beta > > Attachments: HDFS-4733.patch, HDFS-4733.patch, HDFS-4733.patch > > > Continuing with the saga, valid usernames seem to be quite different across > unix distributions. Now running into the case of a setup where they use full > numeric names. > We should make the username pattern configurable via httpfs-site.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644721#comment-13644721
]
Colin Patrick McCabe commented on HDFS-3934:
* properly handle DatanodeID objects where {{getIpAddr()}} and/or
{{getHostName()}} return {{null}}. This fixes the two failing tests.
* don't log an error when an include/exclude file is set to the empty string.
(this is perfectly acceptable; it just means that we don't have such a file.)
* log the name of the include/exclude file we failed to read in our error
message.
> duplicative dfs_hosts entries handled wrong
> ---
>
> Key: HDFS-3934
> URL: https://issues.apache.org/jira/browse/HDFS-3934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Andy Isaacson
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
> HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
> HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
> HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch
>
>
> A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
> hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
> after the NN restarts because {{getDatanodeListForReport}} does not handle
> such a "pseudo-duplicate" correctly:
> # the "Remove any nodes we know about from the map" loop no longer has the
> knowledge to remove the spurious entries
> # the "The remaining nodes are ones that are referenced by the hosts files"
> loop does not do hostname lookups, so does not know that the IP and hostname
> refer to the same host.
> Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
> the JSP output: The *Node* column shows ":50010" as the nodename, with HTML
> markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020";
> title="172.29.97.216:50010">:50010}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3934:
---
Attachment: HDFS-3934.012.patch
> duplicative dfs_hosts entries handled wrong
> ---
>
> Key: HDFS-3934
> URL: https://issues.apache.org/jira/browse/HDFS-3934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Andy Isaacson
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
> HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
> HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
> HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch
>
>
> A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
> hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
> after the NN restarts because {{getDatanodeListForReport}} does not handle
> such a "pseudo-duplicate" correctly:
> # the "Remove any nodes we know about from the map" loop no longer has the
> knowledge to remove the spurious entries
> # the "The remaining nodes are ones that are referenced by the hosts files"
> loop does not do hostname lookups, so does not know that the IP and hostname
> refer to the same host.
> Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
> the JSP output: The *Node* column shows ":50010" as the nodename, with HTML
> markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020";
> title="172.29.97.216:50010">:50010}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4775) Fix HA documentation of ShellCommandFencer for Windows
Arpit Agarwal created HDFS-4775: --- Summary: Fix HA documentation of ShellCommandFencer for Windows Key: HDFS-4775 URL: https://issues.apache.org/jira/browse/HDFS-4775 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 3.0.0 ShellCommandFencer documentation states that it uses bash. Update it to mention that bash is no longer required on Windows and it uses cmd.exe instead. Fencer scripts must be valid cmd.exe scripts. https://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4773) Fix bugs in quota usage updating/computation
[ https://issues.apache.org/jira/browse/HDFS-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4773: Attachment: HDFS-4773.002.patch Update the patch to address Nicholas's comments. > Fix bugs in quota usage updating/computation > > > Key: HDFS-4773 > URL: https://issues.apache.org/jira/browse/HDFS-4773 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-4773.001.patch, HDFS-4773.002.patch > > > 1. FileWithSnapshot#updateQuotaAndCollectBlocks did not consider the scenario > that all the snapshots has been deleted from a snapshot copy of deleted file. > This may lead to a divide-by-0 error. > 2. When computing the quota usage for a WithName node and its subtree, if the > snapshot associated with the WithName node at the time of rename operation > has been deleted, we should compute the quota based on the posterior snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4698) provide client-side metrics for remote reads, local reads, and short-circuit reads
[
https://issues.apache.org/jira/browse/HDFS-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4698:
---
Attachment: HDFS-4698.003.patch
When creating {{DatanodeID}} objects, {{JspHelper}} needs to use
{{InetAddress#getHostAddress}}, not {{InetAddress#toString}}. The latter
returns stuff like {{"keter/127.0.0.1"}} which can't be parsed by
{{NetUtils#createSocketAddr}} (and probably much else).
> provide client-side metrics for remote reads, local reads, and short-circuit
> reads
> --
>
> Key: HDFS-4698
> URL: https://issues.apache.org/jira/browse/HDFS-4698
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.0.4-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-4698.001.patch, HDFS-4698.002.patch,
> HDFS-4698.003.patch
>
>
> We should provide metrics to let clients know how many bytes of data they
> have read remotely, versus locally or via short-circuit local reads. This
> will allow clients to know how well they're doing at bringing the computation
> to the data, which will be useful in evaluating placement policies and
> cluster configurations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4300) TransferFsImage.downloadEditsToStorage should use a tmp file for destination
[
https://issues.apache.org/jira/browse/HDFS-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-4300:
--
Attachment: hdfs-4300-2.patch
Thanks for the review Colin. This newest patch uses a temp filename based on
{{getCurrentTimeMillis}}, which is nice for debugging and I think unique
enough. We additionally do temp file cleanup on startup in
{{SecondaryNameNode#initialize}}.
> TransferFsImage.downloadEditsToStorage should use a tmp file for destination
>
>
> Key: HDFS-4300
> URL: https://issues.apache.org/jira/browse/HDFS-4300
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.2-alpha
>Reporter: Todd Lipcon
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: hdfs-4300-1.patch, hdfs-4300-2.patch
>
>
> Currently, in TransferFsImage.downloadEditsToStorage, we download the edits
> file directly to its finalized path. So, if the transfer fails in the middle,
> a half-written file is left and cannot be distinguished from a correct file.
> So, future checkpoints by the 2NN will fail, since the file is truncated in
> the middle -- but it won't ever download a good copy because it thinks it
> already has the proper file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4776) Backport SecondaryNameNode web ui to branch-1
Tsz Wo (Nicholas), SZE created HDFS-4776: Summary: Backport SecondaryNameNode web ui to branch-1 Key: HDFS-4776 URL: https://issues.apache.org/jira/browse/HDFS-4776 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor The related JIRAs are - HADOOP-3741: SecondaryNameNode has http server on dfs.secondary.http.address but without any contents - HDFS-1728: SecondaryNameNode.checkpointSize is in byte but not MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4776) Backport SecondaryNameNode web ui to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4776: - Attachment: h4776_20130429.patch h4776_20130429.patch > Backport SecondaryNameNode web ui to branch-1 > - > > Key: HDFS-4776 > URL: https://issues.apache.org/jira/browse/HDFS-4776 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h4776_20130429.patch > > > The related JIRAs are > - HADOOP-3741: SecondaryNameNode has http server on > dfs.secondary.http.address but without any contents > - HDFS-1728: SecondaryNameNode.checkpointSize is in byte but not MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4776) Backport SecondaryNameNode web ui to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4776: - Attachment: (was: h4776_20130429.patch) > Backport SecondaryNameNode web ui to branch-1 > - > > Key: HDFS-4776 > URL: https://issues.apache.org/jira/browse/HDFS-4776 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h4776_20130429.patch > > > The related JIRAs are > - HADOOP-3741: SecondaryNameNode has http server on > dfs.secondary.http.address but without any contents > - HDFS-1728: SecondaryNameNode.checkpointSize is in byte but not MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4776) Backport SecondaryNameNode web ui to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4776: - Attachment: h4776_20130429.patch > Backport SecondaryNameNode web ui to branch-1 > - > > Key: HDFS-4776 > URL: https://issues.apache.org/jira/browse/HDFS-4776 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h4776_20130429.patch > > > The related JIRAs are > - HADOOP-3741: SecondaryNameNode has http server on > dfs.secondary.http.address but without any contents > - HDFS-1728: SecondaryNameNode.checkpointSize is in byte but not MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644835#comment-13644835
]
Hadoop QA commented on HDFS-3934:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581007/HDFS-3934.012.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4336//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4336//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4336//console
This message is automatically generated.
> duplicative dfs_hosts entries handled wrong
> ---
>
> Key: HDFS-3934
> URL: https://issues.apache.org/jira/browse/HDFS-3934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Andy Isaacson
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
> HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
> HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
> HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch
>
>
> A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
> hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
> after the NN restarts because {{getDatanodeListForReport}} does not handle
> such a "pseudo-duplicate" correctly:
> # the "Remove any nodes we know about from the map" loop no longer has the
> knowledge to remove the spurious entries
> # the "The remaining nodes are ones that are referenced by the hosts files"
> loop does not do hostname lookups, so does not know that the IP and hostname
> refer to the same host.
> Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
> the JSP output: The *Node* column shows ":50010" as the nodename, with HTML
> markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020";
> title="172.29.97.216:50010">:50010}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4773) Fix bugs in quota usage updating/computation
[ https://issues.apache.org/jira/browse/HDFS-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4773: Attachment: HDFS-4773.003.patch > Fix bugs in quota usage updating/computation > > > Key: HDFS-4773 > URL: https://issues.apache.org/jira/browse/HDFS-4773 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-4773.001.patch, HDFS-4773.002.patch, > HDFS-4773.003.patch > > > 1. FileWithSnapshot#updateQuotaAndCollectBlocks did not consider the scenario > that all the snapshots has been deleted from a snapshot copy of deleted file. > This may lead to a divide-by-0 error. > 2. When computing the quota usage for a WithName node and its subtree, if the > snapshot associated with the WithName node at the time of rename operation > has been deleted, we should compute the quota based on the posterior snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4687) TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
[
https://issues.apache.org/jira/browse/HDFS-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644862#comment-13644862
]
Aaron T. Myers commented on HDFS-4687:
--
+1, the patch looks good to me. I ran this test in isolation with and without
the patch and confirmed that it works as expected.
I'm going to commit this momentarily.
> TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
> --
>
> Key: HDFS-4687
> URL: https://issues.apache.org/jira/browse/HDFS-4687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4687-1.patch
>
>
> Fails fairly often with JDK7.
> {noformat}
> Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.623 sec <<<
> FAILURE!
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 4765 sec <<< FAILURE!
> java.lang.AssertionError: expected:<200> but was:<401>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.junit.Assert.assertEquals(Assert.java:454)
> at
> org.apache.hadoop.hdfs.web.WebHdfsTestUtil.connectAndGetJson(WebHdfsTestUtil.java:78)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:174)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4687) TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
[
https://issues.apache.org/jira/browse/HDFS-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-4687:
-
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this patch to trunk.
Thanks a lot for the contribution, Andrew, and thanks also to Arpit for the
review.
> TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
> --
>
> Key: HDFS-4687
> URL: https://issues.apache.org/jira/browse/HDFS-4687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0
>
> Attachments: hdfs-4687-1.patch
>
>
> Fails fairly often with JDK7.
> {noformat}
> Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.623 sec <<<
> FAILURE!
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 4765 sec <<< FAILURE!
> java.lang.AssertionError: expected:<200> but was:<401>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.junit.Assert.assertEquals(Assert.java:454)
> at
> org.apache.hadoop.hdfs.web.WebHdfsTestUtil.connectAndGetJson(WebHdfsTestUtil.java:78)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:174)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4687) TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
[
https://issues.apache.org/jira/browse/HDFS-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644870#comment-13644870
]
Hudson commented on HDFS-4687:
--
Integrated in Hadoop-trunk-Commit #3690 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3690/])
HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with
JDK7. Contributed by Andrew Wang. (Revision 1477344)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477344
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
> TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
> --
>
> Key: HDFS-4687
> URL: https://issues.apache.org/jira/browse/HDFS-4687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0
>
> Attachments: hdfs-4687-1.patch
>
>
> Fails fairly often with JDK7.
> {noformat}
> Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.623 sec <<<
> FAILURE!
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 4765 sec <<< FAILURE!
> java.lang.AssertionError: expected:<200> but was:<401>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.junit.Assert.assertEquals(Assert.java:454)
> at
> org.apache.hadoop.hdfs.web.WebHdfsTestUtil.connectAndGetJson(WebHdfsTestUtil.java:78)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:174)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}
[jira] [Commented] (HDFS-4698) provide client-side metrics for remote reads, local reads, and short-circuit reads
[
https://issues.apache.org/jira/browse/HDFS-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644876#comment-13644876
]
Hadoop QA commented on HDFS-4698:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581027/HDFS-4698.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4337//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4337//console
This message is automatically generated.
> provide client-side metrics for remote reads, local reads, and short-circuit
> reads
> --
>
> Key: HDFS-4698
> URL: https://issues.apache.org/jira/browse/HDFS-4698
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.0.4-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-4698.001.patch, HDFS-4698.002.patch,
> HDFS-4698.003.patch
>
>
> We should provide metrics to let clients know how many bytes of data they
> have read remotely, versus locally or via short-circuit local reads. This
> will allow clients to know how well they're doing at bringing the computation
> to the data, which will be useful in evaluating placement policies and
> cluster configurations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644880#comment-13644880 ] Suresh Srinivas commented on HDFS-4489: --- Here is NNBench for delete operations (run with 100 threads simultaneously running: ||Opertaions||Elapsed||OpsPerSec||AvgTim|| |10|19243|5196.694902|19| |10|18598|5376.92225|18| |10|17819|5611.987205|17| |10|17953|5570.099705|17| |10|18077|5531.891354|18| |10|17948|5571.651437|17| |10|18080|5530.973451|18| |10|18032|5545.696539|18| |10|18431|5425.641582|18| |10|17735|5638.567804|17| |10|1819|.6 5500|012623 17.7| ||Opertaions||Elapsed||OpsPerSec||AvgTim|| |10|18029|5546.619336|17| |10|18527|5397.527932|18| |10|18164|5505.395287|18| |10|18486|5409.49908|18| |10|18053|5539.24|18| |10|18313|5460.601758|18| |10|18299|5464.779496|18| |10|17878|5593.466831|17| |10|18178|5501.155243|18| |10|18084|5529.750055|18| |10|1820|.1 5494|804057 17.8| > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.0.5-beta > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644880#comment-13644880 ] Suresh Srinivas edited comment on HDFS-4489 at 4/29/13 9:25 PM: Here is NNBench for delete operations (run with 100 threads simultaneously running: ||Opertaions||Elapsed||OpsPerSec||AvgTim|| |10|19243|5196.694902|19| |10|18598|5376.92225|18| |10|17819|5611.987205|17| |10|17953|5570.099705|17| |10|18077|5531.891354|18| |10|17948|5571.651437|17| |10|18080|5530.973451|18| |10|18032|5545.696539|18| |10|18431|5425.641582|18| |10|17735|5638.567804|17| |10|1819|5500.01262|17.7| ||Opertaions||Elapsed||OpsPerSec||AvgTim|| |10|18029|5546.619336|17| |10|18527|5397.527932|18| |10|18164|5505.395287|18| |10|18486|5409.49908|18| |10|18053|5539.24|18| |10|18313|5460.601758|18| |10|18299|5464.779496|18| |10|17878|5593.466831|17| |10|18178|5501.155243|18| |10|18084|5529.750055|18| |10|1820.1|5494.804057|17.7| was (Author: sureshms): Here is NNBench for delete operations (run with 100 threads simultaneously running: ||Opertaions||Elapsed||OpsPerSec||AvgTim|| |10|19243|5196.694902|19| |10|18598|5376.92225|18| |10|17819|5611.987205|17| |10|17953|5570.099705|17| |10|18077|5531.891354|18| |10|17948|5571.651437|17| |10|18080|5530.973451|18| |10|18032|5545.696539|18| |10|18431|5425.641582|18| |10|17735|5638.567804|17| |10|1819|.6 5500|012623 17.7| ||Opertaions||Elapsed||OpsPerSec||AvgTim|| |10|18029|5546.619336|17| |10|18527|5397.527932|18| |10|18164|5505.395287|18| |10|18486|5409.49908|18| |10|18053|5539.24|18| |10|18313|5460.601758|18| |10|18299|5464.779496|18| |10|17878|5593.466831|17| |10|18178|5501.155243|18| |10|18084|5529.750055|18| |10|1820|.1 5494|804057 17.8| > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.0.5-beta > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644884#comment-13644884 ] Aaron T. Myers commented on HDFS-4305: -- Thanks for confirming you're OK with this, Suresh. +1, the latest patch looks good to me. I agree that the test failure seems unrelated. I'm going to commit this momentarily. > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644898#comment-13644898 ] Suresh Srinivas commented on HDFS-4489: --- Summary of results in the tests: # File dreate tests- perform additional reserved name processing, inode map addition and reserved name check. This is where maximum additional work from the patch is being done. #* In the mirco benchmark by just calling create file related methods, the time went from 19235.8 to 19789.2 roughly 2.8% different. This can be further reduced by turning off map to 1.3%. The patch moves splitting paths into components outside the lock. Based on this, further optimizations are possible that improves throughput by reducing the synchronized sections. The end result with that optimizations can make running times much smaller that what it is today. #* I would also point out that, this is a micro benchmark. The % difference observed in this will be dwarfed by RPC times, network round trip time etc. Also the system will spend time on other operations which should not be affected by this patch. # File delete tests - performs reseved name processing and only inode map deletion. #* There very little difference in bench mark results. > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.0.5-beta > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644902#comment-13644902 ] Hudson commented on HDFS-4305: -- Integrated in Hadoop-trunk-Commit #3691 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3691/]) HDFS-4305. Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644907#comment-13644907 ] Suresh Srinivas commented on HDFS-4489: --- Given the above tests, here are all the issues that are brought up: # Introducing incompatible change #* This is not a major incompatibility. As I said earlier, creating file or directory /.reserved is not allowed. That said, this should get into 2.0.5 given its main goal is compatibility. # This patch could be destabilizing #* This patch is adding an Inode map and support for path scheme which allows addressing files by inodes. Most of the code added in this patch is to support the new addressing mechanisms and extensive unit tests associated with it. The regular code path should largely be unaffected by this, with exception of adding and deleting entries in inode map. Please bring up any concerns that I might have overlooked. # Performance impact - based on the results, there is a very little performance impact. I have two options: #* The difference observed in microbenchmarks amounts to much smaller difference in a real system. That too only associated with a few write operations such as create. Hence is it acceptable. #* Make further optimizations to reduce synchronized section size based on the mechanism added in this patch. [~nroberts] if you feel this is important, I will undertake the work of optimizing this. [~daryn] also had expressed interest in it. Not sure if he has the bandwidth. Given this, I would like to merge this in branch-2.0.5. I hope concerns expressed by people are addressed. > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.0.5-beta > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4305: - Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrew. > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4300) TransferFsImage.downloadEditsToStorage should use a tmp file for destination
[
https://issues.apache.org/jira/browse/HDFS-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644936#comment-13644936
]
Hadoop QA commented on HDFS-4300:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581028/hdfs-4300-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4338//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4338//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4338//console
This message is automatically generated.
> TransferFsImage.downloadEditsToStorage should use a tmp file for destination
>
>
> Key: HDFS-4300
> URL: https://issues.apache.org/jira/browse/HDFS-4300
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.2-alpha
>Reporter: Todd Lipcon
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: hdfs-4300-1.patch, hdfs-4300-2.patch
>
>
> Currently, in TransferFsImage.downloadEditsToStorage, we download the edits
> file directly to its finalized path. So, if the transfer fails in the middle,
> a half-written file is left and cannot be distinguished from a correct file.
> So, future checkpoints by the 2NN will fail, since the file is truncated in
> the middle -- but it won't ever download a good copy because it thinks it
> already has the proper file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4773) Fix bugs in quota usage updating/computation
[ https://issues.apache.org/jira/browse/HDFS-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4773: - Hadoop Flags: Reviewed +1 patch looks good. > Fix bugs in quota usage updating/computation > > > Key: HDFS-4773 > URL: https://issues.apache.org/jira/browse/HDFS-4773 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-4773.001.patch, HDFS-4773.002.patch, > HDFS-4773.003.patch > > > 1. FileWithSnapshot#updateQuotaAndCollectBlocks did not consider the scenario > that all the snapshots has been deleted from a snapshot copy of deleted file. > This may lead to a divide-by-0 error. > 2. When computing the quota usage for a WithName node and its subtree, if the > snapshot associated with the WithName node at the time of rename operation > has been deleted, we should compute the quota based on the posterior snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4773) Fix bugs in quota usage updating/computation
[ https://issues.apache.org/jira/browse/HDFS-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4773. -- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) I have committed this. Thanks, Jing! > Fix bugs in quota usage updating/computation > > > Key: HDFS-4773 > URL: https://issues.apache.org/jira/browse/HDFS-4773 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Fix For: Snapshot (HDFS-2802) > > Attachments: HDFS-4773.001.patch, HDFS-4773.002.patch, > HDFS-4773.003.patch > > > 1. FileWithSnapshot#updateQuotaAndCollectBlocks did not consider the scenario > that all the snapshots has been deleted from a snapshot copy of deleted file. > This may lead to a divide-by-0 error. > 2. When computing the quota usage for a WithName node and its subtree, if the > snapshot associated with the WithName node at the time of rename operation > has been deleted, we should compute the quota based on the posterior snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4610. --- Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed +1 for the patch. I committed it to trunk. Thank you Ivan. Thanks to Arpit and Chris for the reviews. > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644983#comment-13644983 ] Suresh Srinivas commented on HDFS-4610: --- Since this patch does not have +1 from Jenkins, I am reverting the patch. > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644986#comment-13644986 ] Hudson commented on HDFS-4610: -- Integrated in Hadoop-trunk-Commit #3692 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3692/]) HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable & FileUtil#canRead/Write/Execute. Contributed by Ivan Mitic. (Revision 1477385) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477385 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-4610: --- > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4610: Status: Patch Available (was: Reopened) > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4774) Backport HDFS-4525 'Provide an API for knowing that whether file is closed or not' to branch 1.1
[
https://issues.apache.org/jira/browse/HDFS-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644991#comment-13644991
]
Ted Yu commented on HDFS-4774:
--
I got the following when running test suite:
{code}
[junit] Test org.apache.hadoop.hdfs.TestFileAppend4 FAILED
[junit] Test org.apache.hadoop.hdfs.TestLargeBlock FAILED (timeout)
[junit] Test org.apache.hadoop.metrics2.impl.TestSinkQueue FAILED
[junit] Test org.apache.hadoop.streaming.TestUlimit FAILED
[junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED
{code}
They all passed when run standalone.
> Backport HDFS-4525 'Provide an API for knowing that whether file is closed or
> not' to branch 1.1
>
>
> Key: HDFS-4774
> URL: https://issues.apache.org/jira/browse/HDFS-4774
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 4774.txt
>
>
> HDFS-4525 compliments lease recovery API which allows user to know whether
> the recovery has completed.
> This JIRA backports the API to branch 1.1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644994#comment-13644994 ] Hudson commented on HDFS-4610: -- Integrated in Hadoop-trunk-Commit #3693 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3693/]) HDFS-4610. Reverting the patch Jenkins build is not run. (Revision 1477396) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477396 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4758: - Attachment: h4758_20140429.patch h4758_20140429.patch: check the snapshottable directories stored in SnapshotManager > Disallow nested snapshottable directories > - > > Key: HDFS-4758 > URL: https://issues.apache.org/jira/browse/HDFS-4758 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h4758_20140426.patch, h4758_20140429.patch > > > Nested snapshottable directories are supported by the current implementation. > However, it seems that there are no good use cases for nested snapshottable > directories. So we disable it for now until someone has a valid use case for > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4777) File creation code in namenode is incorrectly synchronized
Suresh Srinivas created HDFS-4777: - Summary: File creation code in namenode is incorrectly synchronized Key: HDFS-4777 URL: https://issues.apache.org/jira/browse/HDFS-4777 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Blocker FSNamesystem#startFileInternal calls delete. Delete method releases the write lock, making parts of startFileInternal code unintentionally executed without write lock being held. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4777) File creation code in namenode is incorrectly synchronized
[
https://issues.apache.org/jira/browse/HDFS-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645032#comment-13645032
]
Suresh Srinivas commented on HDFS-4777:
---
FSNamesystem calls delete:
{code}
// File exists - must be one of append or overwrite
if (overwrite) {
delete(src, true);
} else {
{code}
and {{deleteInternal()}} release the lock.
> File creation code in namenode is incorrectly synchronized
> --
>
> Key: HDFS-4777
> URL: https://issues.apache.org/jira/browse/HDFS-4777
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.0, 2.0.0-alpha
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Blocker
>
> FSNamesystem#startFileInternal calls delete. Delete method releases the write
> lock, making parts of startFileInternal code unintentionally executed without
> write lock being held.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4687) TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
[
https://issues.apache.org/jira/browse/HDFS-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-4687:
-
Component/s: (was: webhdfs)
test
Priority: Minor (was: Major)
> TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
> --
>
> Key: HDFS-4687
> URL: https://issues.apache.org/jira/browse/HDFS-4687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: hdfs-4687-1.patch
>
>
> Fails fairly often with JDK7.
> {noformat}
> Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.623 sec <<<
> FAILURE!
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 4765 sec <<< FAILURE!
> java.lang.AssertionError: expected:<200> but was:<401>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.junit.Assert.assertEquals(Assert.java:454)
> at
> org.apache.hadoop.hdfs.web.WebHdfsTestUtil.connectAndGetJson(WebHdfsTestUtil.java:78)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:174)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4300) TransferFsImage.downloadEditsToStorage should use a tmp file for destination
[ https://issues.apache.org/jira/browse/HDFS-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4300: -- Attachment: hdfs-4300-3.patch Fix the findbugs warning. > TransferFsImage.downloadEditsToStorage should use a tmp file for destination > > > Key: HDFS-4300 > URL: https://issues.apache.org/jira/browse/HDFS-4300 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Critical > Attachments: hdfs-4300-1.patch, hdfs-4300-2.patch, hdfs-4300-3.patch > > > Currently, in TransferFsImage.downloadEditsToStorage, we download the edits > file directly to its finalized path. So, if the transfer fails in the middle, > a half-written file is left and cannot be distinguished from a correct file. > So, future checkpoints by the 2NN will fail, since the file is truncated in > the middle -- but it won't ever download a good copy because it thinks it > already has the proper file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4758: - Attachment: h4758_20140429b.patch h4758_20140429b.patch: fixes test failures. > Disallow nested snapshottable directories > - > > Key: HDFS-4758 > URL: https://issues.apache.org/jira/browse/HDFS-4758 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h4758_20140426.patch, h4758_20140429b.patch, > h4758_20140429.patch > > > Nested snapshottable directories are supported by the current implementation. > However, it seems that there are no good use cases for nested snapshottable > directories. So we disable it for now until someone has a valid use case for > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[
https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645083#comment-13645083
]
Hadoop QA commented on HDFS-4610:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580927/HDFS-4610.commonfileutils.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4339//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4339//console
This message is automatically generated.
> Move to using common utils FileUtil#setReadable/Writable/Executable and
> FileUtil#canRead/Write/Execute
> --
>
> Key: HDFS-4610
> URL: https://issues.apache.org/jira/browse/HDFS-4610
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HDFS-4610.commonfileutils.2.patch,
> HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch
>
>
> Switch to using common utils described in HADOOP-9413 that work well
> cross-platform.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4777) File creation code in namenode is incorrectly synchronized
[
https://issues.apache.org/jira/browse/HDFS-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645084#comment-13645084
]
Suresh Srinivas commented on HDFS-4777:
---
Digging a little bit deeper, it turns out that my understanding of the lock is
not correct. Here is how it works:
# FSNameystem#startFileInt - acquires writeLock - the hold count for the lock
is 1
# FSNamesystem#deleteInternal - acquires writeLock again - since the thread
holds the lock already the hold count is incremented to 2
# FSNamesystem#deleteInternal - releases writeLock. Hold count is decremented
to 1
# FSNamesystem#deleteInternal - calls logSync(). {color:red}The editlog logSync
now is being done with the FSNamesystem lock held{color}
# FSNamesystem#startFileInt release writeLock. Lock hold count is down to zero
and it is released.
Based on this I am going to update the jira header.
> File creation code in namenode is incorrectly synchronized
> --
>
> Key: HDFS-4777
> URL: https://issues.apache.org/jira/browse/HDFS-4777
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.0, 2.0.0-alpha
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Blocker
>
> FSNamesystem#startFileInternal calls delete. Delete method releases the write
> lock, making parts of startFileInternal code unintentionally executed without
> write lock being held.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4777) File creation with overwrite flag set to true results in logSync holding namesystem lock
[ https://issues.apache.org/jira/browse/HDFS-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4777: -- Summary: File creation with overwrite flag set to true results in logSync holding namesystem lock (was: File creation code in namenode is incorrectly synchronized) > File creation with overwrite flag set to true results in logSync holding > namesystem lock > > > Key: HDFS-4777 > URL: https://issues.apache.org/jira/browse/HDFS-4777 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.0, 2.0.0-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > > FSNamesystem#startFileInternal calls delete. Delete method releases the write > lock, making parts of startFileInternal code unintentionally executed without > write lock being held. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645090#comment-13645090 ] Tsz Wo (Nicholas), SZE commented on HDFS-4489: -- The performance numbers look good. Since the rpc time is not counted, a small percentage difference is nothing. Beside, the Inode ID feature is very useful. It also helps implementing the Snapshot feature. +1 on merging it to branch-2. > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.0.5-beta > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4610: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed the patch to trunk. > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4776) Backport SecondaryNameNode web ui to branch-1
[
https://issues.apache.org/jira/browse/HDFS-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645100#comment-13645100
]
Tsz Wo (Nicholas), SZE commented on HDFS-4776:
--
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] -1 findbugs. The patch appears to introduce 19 new Findbugs
(version 1.3.9) warnings.
{noformat}
The findbugs warnings are not related to the patch.
> Backport SecondaryNameNode web ui to branch-1
> -
>
> Key: HDFS-4776
> URL: https://issues.apache.org/jira/browse/HDFS-4776
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Attachments: h4776_20130429.patch
>
>
> The related JIRAs are
> - HADOOP-3741: SecondaryNameNode has http server on
> dfs.secondary.http.address but without any contents
> - HDFS-1728: SecondaryNameNode.checkpointSize is in byte but not MB.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4760) Update inodeMap after node replacement
[ https://issues.apache.org/jira/browse/HDFS-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4760: Attachment: HDFS-4760.004.patch Update the patch: update the inodeMap in INodeDirectory#replaceSelf and INodeDirectory#replaceChild. > Update inodeMap after node replacement > -- > > Key: HDFS-4760 > URL: https://issues.apache.org/jira/browse/HDFS-4760 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4760.001.patch, HDFS-4760.002.patch, > HDFS-4760.003.patch, HDFS-4760.004.patch > > > Similar with HDFS-4757, we need to update the inodeMap after node > replacement. Because a lot of node replacement happens in the snapshot branch > (e.g., INodeDirectory => INodeDirectoryWithSnapshot, INodeDirectory <=> > INodeDirectorySnapshottable, INodeFile => INodeFileWithSnapshot ...), this > becomes a non-trivial issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645104#comment-13645104 ] Hudson commented on HDFS-4610: -- Integrated in Hadoop-trunk-Commit #3696 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3696/]) HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute. Contrbitued by Ivan Mitic. (Revision 1477427) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477427 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java > Move to using common utils FileUtil#setReadable/Writable/Executable and > FileUtil#canRead/Write/Execute > -- > > Key: HDFS-4610 > URL: https://issues.apache.org/jira/browse/HDFS-4610 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HDFS-4610.commonfileutils.2.patch, > HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch > > > Switch to using common utils described in HADOOP-9413 that work well > cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4765) Permission check of symlink deletion incorrectly throws UnresolvedLinkException
[
https://issues.apache.org/jira/browse/HDFS-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-4765:
--
Attachment: hdfs-4765-1.patch
Patch attached. In the case of delete, we call the permission check without
resolving a symlink if it's the last component.
Verified fail before/pass after behavior with the included unit test.
> Permission check of symlink deletion incorrectly throws
> UnresolvedLinkException
> ---
>
> Key: HDFS-4765
> URL: https://issues.apache.org/jira/browse/HDFS-4765
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.0.3-alpha
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4765-1.patch
>
>
> With permissions enabled, the permission check in {{FSNamesystem#delete}}
> will incorrectly throw an UnresolvedLinkException if the path contains a
> symlink. This leads to FileContext resolving the symlink and instead deleting
> the link target.
> The correct check is to see if the user has write permissions on the parent
> directory of the symlink, e.g.
> {noformat}
> -> % ls -ld symtest
> drwxr-xr-x 2 root root 4096 Apr 26 14:12 symtest
> -> % ls -l symtest
> total 12
> lrwxrwxrwx 1 root root 6 Apr 26 14:12 link -> target
> -rw-r--r-- 1 root root 0 Apr 26 14:11 target
> -> % rm -f symtest/link
> rm: cannot remove `symtest/link': Permission denied
> -> % sudo chown andrew symtest
> -> % rm -f symtest/link
> -> %
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4765) Permission check of symlink deletion incorrectly throws UnresolvedLinkException
[
https://issues.apache.org/jira/browse/HDFS-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-4765:
--
Status: Patch Available (was: Open)
> Permission check of symlink deletion incorrectly throws
> UnresolvedLinkException
> ---
>
> Key: HDFS-4765
> URL: https://issues.apache.org/jira/browse/HDFS-4765
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha, 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4765-1.patch
>
>
> With permissions enabled, the permission check in {{FSNamesystem#delete}}
> will incorrectly throw an UnresolvedLinkException if the path contains a
> symlink. This leads to FileContext resolving the symlink and instead deleting
> the link target.
> The correct check is to see if the user has write permissions on the parent
> directory of the symlink, e.g.
> {noformat}
> -> % ls -ld symtest
> drwxr-xr-x 2 root root 4096 Apr 26 14:12 symtest
> -> % ls -l symtest
> total 12
> lrwxrwxrwx 1 root root 6 Apr 26 14:12 link -> target
> -rw-r--r-- 1 root root 0 Apr 26 14:11 target
> -> % rm -f symtest/link
> rm: cannot remove `symtest/link': Permission denied
> -> % sudo chown andrew symtest
> -> % rm -f symtest/link
> -> %
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4760) Update inodeMap after node replacement
[ https://issues.apache.org/jira/browse/HDFS-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645127#comment-13645127 ] Tsz Wo (Nicholas), SZE commented on HDFS-4760: -- Should INodeReference.setReferredINode(..) also update the map? > Update inodeMap after node replacement > -- > > Key: HDFS-4760 > URL: https://issues.apache.org/jira/browse/HDFS-4760 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4760.001.patch, HDFS-4760.002.patch, > HDFS-4760.003.patch, HDFS-4760.004.patch > > > Similar with HDFS-4757, we need to update the inodeMap after node > replacement. Because a lot of node replacement happens in the snapshot branch > (e.g., INodeDirectory => INodeDirectoryWithSnapshot, INodeDirectory <=> > INodeDirectorySnapshottable, INodeFile => INodeFileWithSnapshot ...), this > becomes a non-trivial issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4760) Update inodeMap after node replacement
[ https://issues.apache.org/jira/browse/HDFS-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4760: Attachment: HDFS-4760.005.patch Yes. setReferredINode is called by replaceSelf and replaceChild. The previous patch missed a scenario in replaceChild. Update the patch to fix. > Update inodeMap after node replacement > -- > > Key: HDFS-4760 > URL: https://issues.apache.org/jira/browse/HDFS-4760 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4760.001.patch, HDFS-4760.002.patch, > HDFS-4760.003.patch, HDFS-4760.004.patch, HDFS-4760.005.patch > > > Similar with HDFS-4757, we need to update the inodeMap after node > replacement. Because a lot of node replacement happens in the snapshot branch > (e.g., INodeDirectory => INodeDirectoryWithSnapshot, INodeDirectory <=> > INodeDirectorySnapshottable, INodeFile => INodeFileWithSnapshot ...), this > becomes a non-trivial issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4751) TestLeaseRenewer#testThreadName flakes
[
https://issues.apache.org/jira/browse/HDFS-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645142#comment-13645142
]
Aaron T. Myers commented on HDFS-4751:
--
Though I think you've correctly identified the issue, I'm not sure that this
patch will definitely address it, since it looks to me like the
{{isFilesBeingWrittenEmpty}} isn't synchronized on DFSClient. Please let me
know if I'm missing something.
> TestLeaseRenewer#testThreadName flakes
> --
>
> Key: HDFS-4751
> URL: https://issues.apache.org/jira/browse/HDFS-4751
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.0.5-beta
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Minor
> Attachments: hdfs-4751-1.patch
>
>
> Seen internally and during upstream trunk builds, error like the following:
> {noformat}
> Error Message:
> Unfinished stubbing detected here: -> at
> org.apache.hadoop.hdfs.TestLeaseRenewer.testThreadName(TestLeaseRenewer.java:197)
> E.g. thenReturn() may be missing. Examples of correct stubbing:
> when(mock.isOk()).thenReturn(true);
> when(mock.isOk()).thenThrow(exception);
> doThrow(exception).when(mock).someVoidMethod(); Hints: 1. missing
> thenReturn() 2. although stubbed methods may return mocks, you cannot inline
> mock creation (mock()) call inside a thenReturn method (see issue 53)
> Stack Trace:
> org.mockito.exceptions.misusing.UnfinishedStubbingException:
> Unfinished stubbing detected here:
> -> at
> org.apache.hadoop.hdfs.TestLeaseRenewer.testThreadName(TestLeaseRenewer.java:197)
> {noformat}
> I believe it's due to mocking while being concurrently accessed by another
> thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4777) File creation with overwrite flag set to true results in logSync holding namesystem lock
[ https://issues.apache.org/jira/browse/HDFS-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4777: -- Attachment: HDFS-4777.patch Changes in the patch: # I am going to make a change I was planning to do for a long time. The attached patch has an assert which fails when editlog sync is called with either FSNamesystem read or write lock held. This should help in catching any accidental introduction of code which calls logSync in critical sections. # This patch also modifies TestFileCreation to create a file with overwrite flag set when a file of that name already exists. This should trigger the logSync assert to fail. In a subsequent patch, I am going to post a fix with which the test failure should not occur. > File creation with overwrite flag set to true results in logSync holding > namesystem lock > > > Key: HDFS-4777 > URL: https://issues.apache.org/jira/browse/HDFS-4777 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.0, 2.0.0-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Attachments: HDFS-4777.patch > > > FSNamesystem#startFileInternal calls delete. Delete method releases the write > lock, making parts of startFileInternal code unintentionally executed without > write lock being held. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4777) File creation with overwrite flag set to true results in logSync holding namesystem lock
[ https://issues.apache.org/jira/browse/HDFS-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4777: -- Status: Patch Available (was: Open) > File creation with overwrite flag set to true results in logSync holding > namesystem lock > > > Key: HDFS-4777 > URL: https://issues.apache.org/jira/browse/HDFS-4777 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.0-alpha, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Attachments: HDFS-4777.patch > > > FSNamesystem#startFileInternal calls delete. Delete method releases the write > lock, making parts of startFileInternal code unintentionally executed without > write lock being held. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4300) TransferFsImage.downloadEditsToStorage should use a tmp file for destination
[
https://issues.apache.org/jira/browse/HDFS-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645151#comment-13645151
]
Hadoop QA commented on HDFS-4300:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581085/hdfs-4300-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFsck
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4340//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4340//console
This message is automatically generated.
> TransferFsImage.downloadEditsToStorage should use a tmp file for destination
>
>
> Key: HDFS-4300
> URL: https://issues.apache.org/jira/browse/HDFS-4300
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.2-alpha
>Reporter: Todd Lipcon
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: hdfs-4300-1.patch, hdfs-4300-2.patch, hdfs-4300-3.patch
>
>
> Currently, in TransferFsImage.downloadEditsToStorage, we download the edits
> file directly to its finalized path. So, if the transfer fails in the middle,
> a half-written file is left and cannot be distinguished from a correct file.
> So, future checkpoints by the 2NN will fail, since the file is truncated in
> the middle -- but it won't ever download a good copy because it thinks it
> already has the proper file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4765) Permission check of symlink deletion incorrectly throws UnresolvedLinkException
[
https://issues.apache.org/jira/browse/HDFS-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645157#comment-13645157
]
Hadoop QA commented on HDFS-4765:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581095/hdfs-4765-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4341//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4341//console
This message is automatically generated.
> Permission check of symlink deletion incorrectly throws
> UnresolvedLinkException
> ---
>
> Key: HDFS-4765
> URL: https://issues.apache.org/jira/browse/HDFS-4765
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.0.3-alpha
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4765-1.patch
>
>
> With permissions enabled, the permission check in {{FSNamesystem#delete}}
> will incorrectly throw an UnresolvedLinkException if the path contains a
> symlink. This leads to FileContext resolving the symlink and instead deleting
> the link target.
> The correct check is to see if the user has write permissions on the parent
> directory of the symlink, e.g.
> {noformat}
> -> % ls -ld symtest
> drwxr-xr-x 2 root root 4096 Apr 26 14:12 symtest
> -> % ls -l symtest
> total 12
> lrwxrwxrwx 1 root root 6 Apr 26 14:12 link -> target
> -rw-r--r-- 1 root root 0 Apr 26 14:11 target
> -> % rm -f symtest/link
> rm: cannot remove `symtest/link': Permission denied
> -> % sudo chown andrew symtest
> -> % rm -f symtest/link
> -> %
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4777) File creation with overwrite flag set to true results in logSync holding namesystem lock
[
https://issues.apache.org/jira/browse/HDFS-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645220#comment-13645220
]
Hadoop QA commented on HDFS-4777:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581106/HDFS-4777.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
org.apache.hadoop.hdfs.TestParallelUnixDomainRead
org.apache.hadoop.hdfs.TestLease
org.apache.hadoop.hdfs.TestLeaseRecovery2
org.apache.hadoop.hdfs.TestLeaseRecovery
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.fs.TestResolveHdfsSymlink
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.TestDFSFinalize
org.apache.hadoop.hdfs.TestDFSStartupVersions
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.TestDataTransferProtocol
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.TestMiniDFSCluster
org.apache.hadoop.fs.viewfs.TestViewFsAtHdfsRoot
org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
org.apache.hadoop.hdfs.TestFetchImage
org.apache.hadoop.hdfs.TestDFSRollback
org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs
org.apache.hadoop.fs.TestHDFSFileContextMainOperations
org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.hdfs.server.namenode.TestINodeFile
org.apache.hadoop.fs.viewfs.TestViewFsHdfs
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.TestHDFSFileSystemContract
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
org.apache.hadoop.hdfs.TestDFSPermission
org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.TestParallelShortCircuitRead
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.web.TestWebHDFS
org.apache.hadoop.hdfs.TestParallelRead
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.serve
[jira] [Updated] (HDFS-4774) Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch 1.1
[ https://issues.apache.org/jira/browse/HDFS-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HDFS-4774: - Summary: Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch 1.1 (was: Backport HDFS-4525 'Provide an API for knowing that whether file is closed or not' to branch 1.1) > Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' > to branch 1.1 > --- > > Key: HDFS-4774 > URL: https://issues.apache.org/jira/browse/HDFS-4774 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 4774.txt > > > HDFS-4525 compliments lease recovery API which allows user to know whether > the recovery has completed. > This JIRA backports the API to branch 1.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4750: - Status: Patch Available (was: Open) > Support NFSv3 interface to HDFS > --- > > Key: HDFS-4750 > URL: https://issues.apache.org/jira/browse/HDFS-4750 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch > > > Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless > integration with client’s file system makes it difficult for users and > impossible for some applications to access HDFS. NFS interface support is one > way for HDFS to have such easy integration. > This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS > client, webHDFS and the NFS interface, HDFS will be easier to access and be > able support more applications and use cases. > We will upload the design document and the initial implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4750: - Attachment: nfs-trunk.patch Uploaded the patch. Before split it into a few JIRAs, I now temporarily put the nfs implementation only under hdfs to make one patch. The test classes are not included. Some subsequent JIRAs will be filed later to address security, stability and other issues. To do some tests with current code: make sure to stop nfs service provided by the platform, keep rpcbind(or portmap) running, 1. start hdfs 2. start nfs gateway using "hadoop nfs3". The nfs gateway has both mountd and nfsd. It has one export HDFS root "/" rw to everyone. 3. mount export to the client, using option such as "-o soft,vers=3,proto=tcp,nolock". Make sure the users on client and server hosts are in synce since nfs gateway uses AUTH_SYS authentication. > Support NFSv3 interface to HDFS > --- > > Key: HDFS-4750 > URL: https://issues.apache.org/jira/browse/HDFS-4750 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch > > > Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless > integration with client’s file system makes it difficult for users and > impossible for some applications to access HDFS. NFS interface support is one > way for HDFS to have such easy integration. > This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS > client, webHDFS and the NFS interface, HDFS will be easier to access and be > able support more applications and use cases. > We will upload the design document and the initial implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645247#comment-13645247
]
Brandon Li commented on HDFS-4750:
--
@Andrew {quote}Would be happy to help with that when you think the code is
ready. {quote}
Thanks! I only did cthon04 basic test(no symlink) with the uploaded patch on
centos. Please feel free to give it a try.
> Support NFSv3 interface to HDFS
> ---
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645250#comment-13645250 ] Zhijie Shen commented on HDFS-4305: --- The patch seems to have broken M/R tests (see MAPREDUCE-5156 and MAPREDUCE-5157). The problem is related to the value of "dfs.namenode.fs-limits.min-block-size". https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3557//testReport/ https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3558//testReport/ > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645264#comment-13645264 ] Aaron T. Myers commented on HDFS-4305: -- Thanks a lot for the report, Zhijie. I've filed an MR JIRA to address this here: MAPREDUCE-5193. I'll upload a patch shortly. > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645267#comment-13645267 ] Zhijie Shen commented on HDFS-4305: --- No problem. Thanks for your quick response:) > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645272#comment-13645272
]
Brandon Li commented on HDFS-4750:
--
@Allen{quote}Any clients besides Linux and Mac OS X? (FWIW: OS X's NFS client
has always been a bit flaky...) Have we thought about YANFS support?{quote}
Weeks ago, did some manual tests with Window NFSv3 client before we changed the
RPC authentication support form AUTH_NULL to AUTH_SYS. We didn't try it again
after the change. Mapping Windows users to Unix users may be needed to test it
again.
We looked a few other NFS implementations. Eventually we decide to implement
one. The major reason is that, the NFS should work around a few HDFS
limitations, and also tightly coupled with HDFS protocols.
> Support NFSv3 interface to HDFS
> ---
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645287#comment-13645287
]
Brandon Li commented on HDFS-4750:
--
@Todd{quote}What is the purpose of putting it in Hadoop proper rather than
proposing it as a separate project (eg in the incubator)? {quote}
What we were thinking was that, as mentioned above, the NFS gateway is tightly
coupled with HDFS protocols. Current code is still controlled at a small size.
Also, some code is so general (e.g., oncrpc implementation) that can be used by
other possible projects.
> Support NFSv3 interface to HDFS
> ---
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645289#comment-13645289
]
Brandon Li commented on HDFS-4750:
--
@Hari{quote}An alternative to consider to support NFS writes is to require
clients do NFS mounts with directio enabled. Directio will bypass client cache
and might alleviate some of the funky behavior.{quote}
Yes, directIO could help reduce the kernel reordered writes. Solaris supports
it using the forcedirectio for mount. Linux seems not have a corresponding
mount option.
> Support NFSv3 interface to HDFS
> ---
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645296#comment-13645296 ] Suresh Srinivas commented on HDFS-4305: --- [~atm] Please mark this as incompatible change. Please also update Release note to describe the nature of incompatibility and how to get around it. > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645297#comment-13645297 ] Suresh Srinivas commented on HDFS-4305: --- One more comment, this should be in the incompatible section in CHANGES.txt. > Add a configurable limit on number of blocks per file, and min block size > - > > Key: HDFS-4305 > URL: https://issues.apache.org/jira/browse/HDFS-4305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 1.0.4, 2.0.4-alpha >Reporter: Todd Lipcon >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: hdfs-4305-1.patch, hdfs-4305-2.patch, hdfs-4305-3.patch > > > We recently had an issue where a user set the block size very very low and > managed to create a single file with hundreds of thousands of blocks. This > caused problems with the edit log since the OP_ADD op was so large > (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To > prevent users from making such mistakes, we should: > - introduce a configurable minimum block size, below which requests are > rejected > - introduce a configurable maximum number of blocks per file, above which > requests to add another block are rejected (with a suitably high default as > to not prevent legitimate large files) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
Devaraj Das created HDFS-4778: - Summary: Invoke getPipeline in the chooseTarget implementation that has favoredNodes Key: HDFS-4778 URL: https://issues.apache.org/jira/browse/HDFS-4778 Project: Hadoop HDFS Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. There are some other minor issues discovered. Will fix all of them in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645314#comment-13645314 ] Devaraj Das commented on HDFS-4778: --- Has a couple of fixes. 1. I had missed invoking getPipeline on the nodes in the patch on HDFS-2576. 2. I came to realize that the getBlockLocations call may not returns a BlockLocation with all the hosts filled in. It may take some time for the namenode to give a BlockLocation with the complete set of hosts. The patch puts in loops around those calls. 3. Fixes a javadoc. > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4778: -- Attachment: hdfs-4778-1.patch The patch. > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Attachments: hdfs-4778-1.patch > > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4778: -- Fix Version/s: 2.0.5-beta Status: Patch Available (was: Open) > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-4778-1.patch > > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Resolution: Fixed Status: Resolved (was: Patch Available) Resolving this patch. Will open another jira for branch-1 > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
Devaraj Das created HDFS-4779: - Summary: Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) Key: HDFS-4779 URL: https://issues.apache.org/jira/browse/HDFS-4779 Project: Hadoop HDFS Issue Type: Bug Reporter: Devaraj Das -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
[ https://issues.apache.org/jira/browse/HDFS-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4779: -- Fix Version/s: 1.3.0 > Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) > - > > Key: HDFS-4779 > URL: https://issues.apache.org/jira/browse/HDFS-4779 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das > Fix For: 1.3.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
[ https://issues.apache.org/jira/browse/HDFS-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4779: -- Attachment: hdfs-2576-branch-1-wip.patch Patch that has not seen much testing. > Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) > - > > Key: HDFS-4779 > URL: https://issues.apache.org/jira/browse/HDFS-4779 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das > Fix For: 1.3.0 > > Attachments: hdfs-2576-branch-1-wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
[ https://issues.apache.org/jira/browse/HDFS-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645319#comment-13645319 ] Devaraj Das commented on HDFS-4779: --- The patch includes the fixes in HDFS-4778 > Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) > - > > Key: HDFS-4779 > URL: https://issues.apache.org/jira/browse/HDFS-4779 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das > Fix For: 1.3.0 > > Attachments: hdfs-2576-branch-1-wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645322#comment-13645322
]
Brandon Li commented on HDFS-4750:
--
@Han{quote}..not having understood the description in the pdf that "Hadoop+FUSE
could be used to provide an NFS interface for HDFS. However it has many known
problems and limitations."{quote}
For example, FUSE is not inode based like NFS. FUSE usually uses path to
generate the NFS file handle. Its path-handle mapping can make the host run out
of memory. Even it can work around the memory problem, it could have
correctness issue. FUSE may not be aware that a file's path has been changed by
other means(e.g., hadoop CLI). If FUSE is used on the client side, each NFS
client has to install a client component which runs only on Linux so far. ...
> Support NFSv3 interface to HDFS
> ---
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645326#comment-13645326
]
Hadoop QA commented on HDFS-4750:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581119/nfs-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1371 javac
compiler warnings (more than the trunk's current 1366 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 19 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4343//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4343//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4343//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4343//console
This message is automatically generated.
> Support NFSv3 interface to HDFS
> ---
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf, nfs-trunk.patch
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
