[jira] Updated: (HDFS-1338) Improve TestDFSIO
[ https://issues.apache.org/jira/browse/HDFS-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1338: - Attachment: TestDFSIOparser_y0.20.patch TestDFSIOparser_y0.20.patch: a small program to parse TestDFSIO output. Improve TestDFSIO - Key: HDFS-1338 URL: https://issues.apache.org/jira/browse/HDFS-1338 Project: Hadoop HDFS Issue Type: Improvement Reporter: Arun C Murthy Attachments: TestDFSIOparser_y0.20.patch Currently the read test in TestDFSIO benchmark just opens a large side file and measures the read performance. The MR scheduler has no opportunity to do *any* optimization for the TestDFSIO MR application. The side-effect of this is that it is *very* hard to do any meaningful analysis of the results of the benchmark i.e. to check if node-local or rack-local or off-switch read performance improved/degraded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1338) Improve TestDFSIO
[ https://issues.apache.org/jira/browse/HDFS-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1338: - Component/s: test Improve TestDFSIO - Key: HDFS-1338 URL: https://issues.apache.org/jira/browse/HDFS-1338 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Arun C Murthy Attachments: TestDFSIOparser_y0.20.patch Currently the read test in TestDFSIO benchmark just opens a large side file and measures the read performance. The MR scheduler has no opportunity to do *any* optimization for the TestDFSIO MR application. The side-effect of this is that it is *very* hard to do any meaningful analysis of the results of the benchmark i.e. to check if node-local or rack-local or off-switch read performance improved/degraded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1338) Improve TestDFSIO
[ https://issues.apache.org/jira/browse/HDFS-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898331#action_12898331 ] Rajesh Balamohan commented on HDFS-1338: I am on vacation from 13-Aug - 15-Aug. I do not have internet access during this time. Please check with my manager sriguru@ for any urgent issues. ~Rajesh.B Improve TestDFSIO - Key: HDFS-1338 URL: https://issues.apache.org/jira/browse/HDFS-1338 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Arun C Murthy Attachments: TestDFSIOparser_y0.20.patch Currently the read test in TestDFSIO benchmark just opens a large side file and measures the read performance. The MR scheduler has no opportunity to do *any* optimization for the TestDFSIO MR application. The side-effect of this is that it is *very* hard to do any meaningful analysis of the results of the benchmark i.e. to check if node-local or rack-local or off-switch read performance improved/degraded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1335) HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode
[ https://issues.apache.org/jira/browse/HDFS-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898345#action_12898345 ] Doug Cutting commented on HDFS-1335: Thanks for elaborating. This patch seems fine: it's simple and does not alone introduce significant risk. However subsequent changes to ProtocolCompatible will need to be carefully reviewed and tested. We don't yet have a testing framework to do that. So we're delaying that work (interoperability testing of different versions) until we actually have two versions that are meant to interoperate. This is a technical debt will be due in full the first time ProtocolCompatible is altered, I think. HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode - Key: HDFS-1335 URL: https://issues.apache.org/jira/browse/HDFS-1335 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.22.0 Reporter: Hairong Kuang Assignee: Hairong Kuang Fix For: 0.22.0 Attachments: hdfsRpcVersion.patch The idea is that for getProtocolVersion, NameNode checks if the client and server versions are compatible if the server version is greater than the client version. If no, throws a VersionIncompatible exception; otherwise, returns the server version. On the dfs client side, when creating a NameNode proxy, catches the VersionMismatch exception and then checks if the client version and the server version are compatible if the client version is greater than the server version. If not compatible, throws exception VersionIncomptible; otherwise, records the server version and continues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898431#action_12898431 ] Jitendra Nath Pandey commented on HDFS-1340: The path was manually tested on trunk in both secure and insecure mode. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898430#action_12898430 ] Jitendra Nath Pandey commented on HDFS-1340: ant test was run manually. All tests pass except TestHDFSTrash which is unrelated. test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1301) TestHDFSProxy need to use server side conf for ProxyUser stuff.
[ https://issues.apache.org/jira/browse/HDFS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1301: - Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Resolution: Fixed committed to trunk. TestHDFSProxy need to use server side conf for ProxyUser stuff. --- Key: HDFS-1301 URL: https://issues.apache.org/jira/browse/HDFS-1301 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1301-1.patch, HDFS-1301-BP20-1.patch, HDFS-1301-BP20.patch, HDFS-1301.patch currently TestHdfsProxy sets hadoop.proxyuser.USER.groups in local copy of configuration. But ProxyUsers only looks at the server side config. For test we can uses static method in ProxyUsers to load the config. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1340: --- Attachment: HDFS-1340.2.patch New patch..added a unit test. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898443#action_12898443 ] Devaraj Das commented on HDFS-1340: --- Can you please check both the code paths in the unit test? Looks fine otherwise A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1340: --- Attachment: HDFS-1340.3.patch Updated unit test. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898447#action_12898447 ] Devaraj Das commented on HDFS-1340: --- I understand that security remains enabled in the added unit test since a previous test sets it to kerberos. But it is safer to explicitly enable it in the beginning of the test you added. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1340: --- Attachment: HDFS-1340.4.patch Updated the unit test addressing the comment. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch, HDFS-1340.4.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1340: --- Status: Open (was: Patch Available) A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch, HDFS-1340.4.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1340: --- Status: Patch Available (was: Open) A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch, HDFS-1340.4.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1340: -- Status: Resolved (was: Patch Available) Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Jitendra! A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch, HDFS-1340.4.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1340: --- Attachment: HDFS-1340.y20.1.patch Patch for y20 branch. A null delegation token is appended to the url if security is disabled when browsing filesystem. Key: HDFS-1340 URL: https://issues.apache.org/jira/browse/HDFS-1340 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch, HDFS-1340.4.patch, HDFS-1340.y20.1.patch When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1341) Provide a mechanism to run functional tests on a real cluster
Provide a mechanism to run functional tests on a real cluster - Key: HDFS-1341 URL: https://issues.apache.org/jira/browse/HDFS-1341 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 0.22.0 Reporter: Konstantin Boudnik With Herriot framework (implemented by HADOOP-6332 at all) in place it'd be great to have a way to run on a real cluster all (or as many as possible) functional tests using MiniDFSCluster at the moment. This might possible be accomplished in two ways: - by runtime dependency injection to specify underlying cluster abstraction to be chosen (MiniDFSCluster mock or a proxy for real cluster provided by Herriot library) - by extending MiniDFSCluster to become a transparent adapter between a test and Herriot's real cluster proxy. Having this in place will tremendously benefit overall quality of HDFS (and potentially all Hadoop) be reusing existing functional testbase for system testing. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1335) HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode
[ https://issues.apache.org/jira/browse/HDFS-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898503#action_12898503 ] dhruba borthakur commented on HDFS-1335: This approach sounds good. Do we need to implement the same mechanisms for DatanodeProtocol and NameNodeProtocol as well? HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode - Key: HDFS-1335 URL: https://issues.apache.org/jira/browse/HDFS-1335 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.22.0 Reporter: Hairong Kuang Assignee: Hairong Kuang Fix For: 0.22.0 Attachments: hdfsRpcVersion.patch The idea is that for getProtocolVersion, NameNode checks if the client and server versions are compatible if the server version is greater than the client version. If no, throws a VersionIncompatible exception; otherwise, returns the server version. On the dfs client side, when creating a NameNode proxy, catches the VersionMismatch exception and then checks if the client version and the server version are compatible if the client version is greater than the server version. If not compatible, throws exception VersionIncomptible; otherwise, records the server version and continues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1342) expose DFSOutputStream.getNumCurrentReplicas() in libhdfs
expose DFSOutputStream.getNumCurrentReplicas() in libhdfs - Key: HDFS-1342 URL: https://issues.apache.org/jira/browse/HDFS-1342 Project: Hadoop HDFS Issue Type: New Feature Components: contrib/libhdfs Reporter: sam rash Assignee: sam rash Priority: Minor DFSOutputStream exposes the number of writers in a pipeline. We should make this callable from libhdfs -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
