[jira] [Created] (HADOOP-8760) windows 7 : error accessing journals from distributed cache
Haytham created HADOOP-8760: --- Summary: windows 7 : error accessing journals from distributed cache Key: HADOOP-8760 URL: https://issues.apache.org/jira/browse/HADOOP-8760 Project: Hadoop Common Issue Type: Bug Components: filecache Affects Versions: 0.20.2 Environment: Widnows 7 x64. Reporter: Haytham error accessing journals from distributed cache: This is because in org.apache.hadoop.filecache.TrackerDistributedCacheManager downloadCacheObject In < if (!localFs.rename(workDir, finalDir)) > , the file rename in windows failed and made instead a copy of the file. Thus, the required cache file does not exist in the required folder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447676#comment-13447676 ] Robert Joseph Evans commented on HADOOP-8758: - One really nice use case that Daryn Sharp has talked a lot about is testing. If you could easily replace Kerberos with a mock all of the unit tests can be run with security enabled. There are a lot of security bugs that creep in because we cannot test with security on all the time. It also would allow us to use tokens when "security" is off. This would reduce the number of code paths dramatically which in turn is will improve the coverage of the unit tests. I don't know of many people that really will want to replace Kerberos entirely in their grids some might, but at a minimum from a testing and stability perspective it is very interesting. > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
Philip Zeyliger created HADOOP-8761: --- Summary: Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes) Key: HADOOP-8761 URL: https://issues.apache.org/jira/browse/HADOOP-8761 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Philip Zeyliger Priority: Minor Trivial patch attached corrects the usage information. Stat.java calls FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Zeyliger updated HADOOP-8761: Assignee: Philip Zeyliger Status: Patch Available (was: Open) Submitting patch for Hudson. No tests were added. > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Zeyliger updated HADOOP-8761: Attachment: HADOOP-8761.patch.txt Trivial patch attached. > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447735#comment-13447735 ] Hadoop QA commented on HADOOP-8761: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543672/HADOOP-8761.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.cli.TestCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1398//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1398//console This message is automatically generated. > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8762) Mark container-provided dependencies with 'provided' scope
Tom White created HADOOP-8762: - Summary: Mark container-provided dependencies with 'provided' scope Key: HADOOP-8762 URL: https://issues.apache.org/jira/browse/HADOOP-8762 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.0-alpha Reporter: Tom White For example the Tomcat and Jetty dependencies should be 'provided' since they are provided by the container (i.e. the Hadoop installation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447751#comment-13447751 ] Owen O'Malley commented on HADOOP-8758: --- This is reasonable, although it isn't pluggable token implementations as much as adding new authentication mechanisms. (In other words, you shouldn't change the token mechanisms like the secret manager, but add an alternative to Kerberos and delegation tokens.) I really wish we had a better SASL shared secret implementation than one that depends on MD5, whose strength is already pretty questionable. Clearly that is a related, but different issue. I think a fruitful direction would be to figure out how to authenticate the HTTP connection that fetches delegation tokens. The HTTP authentication is already pluggable using servlet filters. Further, we've already done proprietary and spnego security filters, so I suspect that it is flexible enough for most purposes. It doesn't require any sasl, rpc, or client changes and doesn't break backwards compatibility. Would that work for your use case? > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1344#comment-1344 ] Daryn Sharp commented on HADOOP-8758: - As Bobby mentioned, and I've mentioned to Owen offline, I've really been wanting to configure the login context to use an auth other than kerberos, yet still use a secret manager to generate and validate tokens. Many of the tough issues I've had to debug may be caught by the test suites if simple auth used tokens too. One problem to solve is that kerberos has crept up above the login context's abstraction, ex. the UGI. Although it's technically possible to use a different auth, there's a lot of places where checks for kerberos is hardcoded. > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447810#comment-13447810 ] Daryn Sharp commented on HADOOP-8761: - I think the correct behavior should be to match the usage. The unix stat uses %b for blocks, and %s for size in bytes. > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447840#comment-13447840 ] Andy Isaacson commented on HADOOP-8761: --- bq. I think the correct behavior should be to match the usage. The unix stat uses %b for blocks, and %s for size in bytes. I agree that adding %s for size in bytes is a good idea, but changing the semantics of %b is an incompatible change, which seems risky. OTOH, I think it would be potentially very useful to expose "number of blocks in this file". Tangentially, the documentation also fails to describe %F. > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8755) Print thread dump when tests fail due to timeout
[ https://issues.apache.org/jira/browse/HADOOP-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447843#comment-13447843 ] Andrey Klochkov commented on HADOOP-8755: - Hi Aaron, You're right - changing this timeout wouldn't help. There are 2 timeouts for test execution time: one in surefire and another is in junit. Surefire just kills a child process when timeout is exceeded, and this patch doesn't handle this. What's handled is if a test method is annotated with @Test and the timeout parameter is given, then junit will fail the test and thread dump will be printed. TestFileAppend4 is an example of a test providing timeout parameter. When testing the patch I reduced timeout in TestFileAppend4 and validated that thread dump is presented in the test output file. AFAIK we can't really do anything with the surefire timeout. Still we may have thread dumps printed for all tests in case of timeouts if we introduce a default timeout for all tests on the junit level. I guess it is doable with a custom surefire provider for junit, but I'm not sure we really need this. What do you think? > Print thread dump when tests fail due to timeout > - > > Key: HADOOP-8755 > URL: https://issues.apache.org/jira/browse/HADOOP-8755 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: 1.0.3, 0.23.1, 2.0.0-alpha >Reporter: Andrey Klochkov >Assignee: Andrey Klochkov > Attachments: HDFS-3762-branch-0.23.patch, HDFS-3762.patch, > HDFS-3762.patch, HDFS-3762.patch, HDFS-3762.patch, HDFS-3762.patch > > > When a test fails due to timeout it's often not clear what is the root cause. > See HDFS-3364 as an example. > We can print dump of all threads in this case, this may help finding causes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8761: --- Target Version/s: 2.2.0-alpha Affects Version/s: 2.0.0-alpha Targeting for 2.2.0. Philip, it looks like the TestCLI failure is related, probably because it's expecting some specific help text. > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.0.0-alpha >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8763) Set group owner on Windows failed
Chuan Liu created HADOOP-8763: - Summary: Set group owner on Windows failed Key: HADOOP-8763 URL: https://issues.apache.org/jira/browse/HADOOP-8763 Project: Hadoop Common Issue Type: Bug Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor Fix For: 1-win RawLocalFileSystem.setOwner() method may incorrectly set the group owner of a file on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8754) remove all the RPC.getServer() variants
[ https://issues.apache.org/jira/browse/HADOOP-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447929#comment-13447929 ] Brandon Li commented on HADOOP-8754: OK. Let's be a bit conservative, deprecate them first. Modifying the title accordingly... > remove all the RPC.getServer() variants > --- > > Key: HADOOP-8754 > URL: https://issues.apache.org/jira/browse/HADOOP-8754 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li >Priority: Minor > > In HADOOP-8736, a Builder is introduced to replace all the getServer() > variants. This JIRA is to delete all the getServer() variants once all the > other components replaced this method with the Builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8754) Deprecate all the RPC.getServer() variants
[ https://issues.apache.org/jira/browse/HADOOP-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HADOOP-8754: --- Summary: Deprecate all the RPC.getServer() variants (was: remove all the RPC.getServer() variants) > Deprecate all the RPC.getServer() variants > -- > > Key: HADOOP-8754 > URL: https://issues.apache.org/jira/browse/HADOOP-8754 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li >Priority: Minor > > In HADOOP-8736, a Builder is introduced to replace all the getServer() > variants. This JIRA is to delete all the getServer() variants once all the > other components replaced this method with the Builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8763) Set group owner on Windows failed
[ https://issues.apache.org/jira/browse/HADOOP-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu updated HADOOP-8763: -- Attachment: HADOOP-8763-branch-1-win.patch Attach a patch. > Set group owner on Windows failed > - > > Key: HADOOP-8763 > URL: https://issues.apache.org/jira/browse/HADOOP-8763 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu >Priority: Minor > Fix For: 1-win > > Attachments: HADOOP-8763-branch-1-win.patch > > > RawLocalFileSystem.setOwner() method may incorrectly set the group owner of a > file on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8762) Mark container-provided dependencies with 'provided' scope
[ https://issues.apache.org/jira/browse/HADOOP-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8762: -- Attachment: HADOOP-8762.patch This patch changes container dependencies to 'provided' for Common and HDFS. With this change I could remove the excludes from hadoop-client without bringing in any extra dependencies (in fact hadoop-client was incorrectly pulling in Jetty via HDFS). > Mark container-provided dependencies with 'provided' scope > -- > > Key: HADOOP-8762 > URL: https://issues.apache.org/jira/browse/HADOOP-8762 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.0.0-alpha >Reporter: Tom White > Attachments: HADOOP-8762.patch > > > For example the Tomcat and Jetty dependencies should be 'provided' since they > are provided by the container (i.e. the Hadoop installation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[ https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447989#comment-13447989 ] Daryn Sharp commented on HADOOP-8761: - True, it's incompatible yet correct behavior. I guess our options are codify the bad behavior forever, or fix it now. I lean toward the latter since there's enough churn in 2.x that this will be a minor inconvenience for people to adjust. What do the other watchers/lurkers think? > Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be > bytes) > > > Key: HADOOP-8761 > URL: https://issues.apache.org/jira/browse/HADOOP-8761 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.0.0-alpha >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: HADOOP-8761.patch.txt > > > Trivial patch attached corrects the usage information. Stat.java calls > FileStatus.getLen(), which is most definitely the file size in bytes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448057#comment-13448057 ] eric baldeschwieler commented on HADOOP-8758: - I really like the idea of code path elimination! What about the use case of a proxy, where all external work comes via a gateway that already has authenticated the user via LDAP or some other mechanism? If all user interaction comes from this central point, can we hook that up to the existing token mechanism and maintain security? - Gateway -> HDFS - Gateway -> MR -> HDFS - Gateway -> MR -> HBase -> HDFS - Gateway -> Oozie -> MR -> Hive -> (HCat & MR) -> HDFS > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7682) taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448072#comment-13448072 ] Todd Fast commented on HADOOP-7682: --- Thanks, Joshua! Your workaround got us running locally on Hadoop 1.0.3 without any issues. Here is a GitHub repository for the source (as well as a pre-built JAR): https://github.com/congainc/patch-hadoop_7682-1.0.x-win > taskTracker could not start because "Failed to set permissions" to "ttprivate > to 0700" > -- > > Key: HADOOP-7682 > URL: https://issues.apache.org/jira/browse/HADOOP-7682 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 1.0.1 > Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, > jdk1.6.0_05 >Reporter: Magic Xie > > ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because > java.io.IOException:Failed to set permissions of > path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 > at > org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) > at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) > at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) > Since hadoop0.20.203 when the TaskTracker initialize, it checks the > permission(TaskTracker Line 624) of > (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, > > org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) > call setPermission(Line 481) to deal with it, setPermission works fine on > *nx, however,it dose not alway works on windows. > setPermission call setReadable of Java.io.File in the line 498, but according > to the Table1 below provided by oracle,setReadable(false) will always return > false on windows, the same as setExecutable(false). > http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ > is it cause the task tracker "Failed to set permissions" to "ttprivate to > 0700"? > Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
[
https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-8756:
-
Description:
* We use {{System.loadLibrary("snappy")}} from the Java side. However in
libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like
the System.loadLibrary nonsense is completely redundant-- we would only need
this if we planned on accessing libsnappy methods directly from Java (via the
'native' keyword)- - which we don't, and can't, since snappy is not JNI. At
the end of the day, snappy is only accessible through libhadoop.so, so I think
snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's
easier to diagnose configuration issues.
was:
* We use {{System.loadLibrary("snappy")}} from the Java side. However in
libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like
the System.loadLibrary nonsense is completely redundant-- we would only need
this if we planned on accessing libsnappy methods directly from Java (via the
'native' keyword)- - which we don't, and can't, since snappy is not JNI. At
the end of the day, snappy is only accessible through libhadoop.so, so I think
snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's
easier to diagnose configuration issues.
* It seems like the static initialization code in the LoadSnappy.java class
references the static initialization code in NativeCodeLoader. Basically, if
the init method in NativeCodeLoader doesn't run before the init method in
LoadSnappy, there could be problems. We should avoid having this kind of issue.
> libsnappy loader issues
> ---
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
>
> * We use {{System.loadLibrary("snappy")}} from the Java side. However in
> libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like
> the System.loadLibrary nonsense is completely redundant-- we would only need
> this if we planned on accessing libsnappy methods directly from Java (via the
> 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At
> the end of the day, snappy is only accessible through libhadoop.so, so I
> think snappy-related issues should be handled there.
> * We should log the search path(s) we use for {{libsnappy.so}}, so that it's
> easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448081#comment-13448081 ] Alejandro Abdelnur commented on HADOOP-8758: A recurrent pattern I've seen in secure deployments is that in many cases they don't want to expose Kerberos outside of the cluster/gateway machines. Specially on the browsers. Due to internal IT rules, they don't want to kerberize enduser computers (many times laptops). Given this, I think we should consider decoupling cluster/inter-services authentication (based on Kerberos) from user facing authentication (based on something more palatable like LDAP or AD). Regarding the current token mechanism, there are several classes and sub-classses for different components, I wonder why we need all of them instead using a single implementation. Finally, on Eric's comment, I think most services already provide proxyuser support for propagating caller credentials. JT/NN/Oozie/HttpFS do. May not the ideal mechanism but it works fine. > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8761) Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be bytes)
[
https://issues.apache.org/jira/browse/HADOOP-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448102#comment-13448102
]
Andy Isaacson commented on HADOOP-8761:
---
bq. True, it's incompatible yet correct behavior. I guess our options are
codify the bad behavior forever, or fix it now.
Since this is client-side code, we could do something like
# in 2.1.0, add %s and leave %b giving bytes, with a stderr message that %b
will change in a future release.
# in 2.2.0, change %b to give blocks and remove the stderr message.
bq. True, it's incompatible yet correct behavior.
I'm a little skeptical that "correct" with respect to POSIX stat(2) is
significant here. The "blocks" used in {{struct stat}} are very different from
the "blocks" in HDFS; POSIX blocks are fixed size, atomically written [1], and
the block size is a legacy feature (hardcoded to 512 bytes which is not the
underlying size of anything anymore). By contrast HDFS blocks are variable
size, nonatomic, and are much larger than POSIX blocks.
I agree that exposing the blocksize (and the blockcount) is a pretty valuable
feature, but there's a lot of caveats. Just off the top of my head: a single
file can have multiple blocksizes.
[1] blocks aren't guaranteed to be atomic by the POSIX spec AFAIK, but as a
practical matter modern implementations are atomic at some blocksize between
512 and 4096 bytes.
> Help for FsShell's Stat incorrectly mentions "file size in blocks" (should be
> bytes)
>
>
> Key: HADOOP-8761
> URL: https://issues.apache.org/jira/browse/HADOOP-8761
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.0.0-alpha
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HADOOP-8761.patch.txt
>
>
> Trivial patch attached corrects the usage information. Stat.java calls
> FileStatus.getLen(), which is most definitely the file size in bytes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448161#comment-13448161 ] Daryn Sharp commented on HADOOP-8758: - @Eric: Today, hadoop's rpc security as all predicated on kerberos so the gateway daemon would require kerberos auth (keytab) to communicate with other hadoop daemons. The NN and other daemons issuing tokens need to be configured to trust the gateway daemon as a quasi-root user so it can pretend to be other users. This is the "proxy token" support currently used by oozie. The oozie server is kerberos auth-ed and the other daemons allow oozie to request tokens to masquerade as other users. Proxy tokens provide hadoop with rudimentary support for non-kerberos auth, albeit in a questionably secure way. If I could successfully exploit an oozie server or gateway and gain access to its keytab, then I could access almost anyone's data. It would be more secure if the hadoop daemons directly support other auth methods to avoid having quasi-root users configured within the system. bq. I really like the idea of code path elimination! Me too, especially in this case! Less security code paths by always using tokens for authz would be a dream come true. The code would be streamlined, and devs w/o access to a secure cluster would know if they broke tokens, regardless of whether kerberos, ldap, etc, or no security is enabled. > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8749) HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.
[ https://issues.apache.org/jira/browse/HADOOP-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated HADOOP-8749: - Attachment: HADOOP-8749_rev2.patch Thanks Todd, ATM and Steve! Here is the updated patch. I have added a test case to cover when a relative path is used in the XInclude. > HADOOP-8031 changed the way in which relative xincludes are handled in > Configuration. > - > > Key: HADOOP-8749 > URL: https://issues.apache.org/jira/browse/HADOOP-8749 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: HADOOP-8749.patch, HADOOP-8749_rev2.patch > > > The patch from HADOOP-8031 changed the xml parsing to use > DocumentBuilder#parse(InputStream uri.openStream()) instead of > DocumentBuilder#parse(String uri.toString()).I looked into the implementation > of javax.xml.parsers.DocumentBuilder and org.xml.sax.InputSource and there is > a difference when the DocumentBuilder parse(String) method is used versus > parse(InputStream). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
[
https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-8756:
-
Description:
We use {{System.loadLibrary("snappy")}} from the Java side. However in
libhadoop, we use {{dlopen}} to open libsnappy.so dynamically.
System.loadLibrary uses {{java.library.path}} to resolve libraries, and
{{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries.
Because of this, the two library loading functions can be at odds.
We should fix this so we only load the library once, preferably using the
standard Java {{java.library.path}}.
We should also log the search path(s) we use for {{libsnappy.so}} when loading
fails, so that it's easier to diagnose configuration issues.
was:
* We use {{System.loadLibrary("snappy")}} from the Java side. However in
libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like
the System.loadLibrary nonsense is completely redundant-- we would only need
this if we planned on accessing libsnappy methods directly from Java (via the
'native' keyword)- - which we don't, and can't, since snappy is not JNI. At
the end of the day, snappy is only accessible through libhadoop.so, so I think
snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's
easier to diagnose configuration issues.
> libsnappy loader issues
> ---
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in
> libhadoop, we use {{dlopen}} to open libsnappy.so dynamically.
> System.loadLibrary uses {{java.library.path}} to resolve libraries, and
> {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve
> libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the
> standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when
> loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
Trevor Robinson created HADOOP-8764:
---
Summary: CMake: HADOOP-8737 broke ARM build
Key: HADOOP-8764
URL: https://issues.apache.org/jira/browse/HADOOP-8764
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.2.0-alpha
Environment: Apache Maven 3.0.4
Maven home: /usr/share/maven
Java version: 1.7.0_06, vendor: Oracle Corporation
Java home: /usr/lib/jvm/jdk1.7.0_06/jre
Default locale: en_US, platform encoding: ISO-8859-1
OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
Reporter: Trevor Robinson
ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
which reports values like "armv7l" for the ARMv7 architecture. However, the
OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trevor Robinson updated HADOOP-8764:
Attachment: HADOOP-8764.patch
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trevor Robinson updated HADOOP-8764:
Assignee: Trevor Robinson
Status: Patch Available (was: Open)
Added processor architecture decode case for ARM similar to x86:
{code}
ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")
SET(_java_libarch "arm")
{code}
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448447#comment-13448447
]
Hadoop QA commented on HADOOP-8764:
---
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543798/HADOOP-8764.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1399//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1399//console
This message is automatically generated.
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448463#comment-13448463
]
Eli Collins commented on HADOOP-8764:
-
+1 lgtm
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448465#comment-13448465
]
Eli Collins commented on HADOOP-8764:
-
Btw I created a jenkins job to run on the Apache ARM machines to capture future
breakages:
https://builds.apache.org/job/Hadoop-trunk-ARM/
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HADOOP-8764:
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2. Thanks Trevor!
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448468#comment-13448468
]
Hudson commented on HADOOP-8764:
Integrated in Hadoop-Common-trunk-Commit #2682 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2682/])
HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor
Robinson (Revision 1380984)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380984
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8737) cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
[
https://issues.apache.org/jira/browse/HADOOP-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448469#comment-13448469
]
Hudson commented on HADOOP-8737:
Integrated in Hadoop-Common-trunk-Commit #2682 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2682/])
HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor
Robinson (Revision 1380984)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380984
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
> cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
> --
>
> Key: HADOOP-8737
> URL: https://issues.apache.org/jira/browse/HADOOP-8737
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8737.001.patch, HADOOP-8737.002.patch
>
>
> We should always use the {{libjvm.so}}, {{jni.h}}, and {{jni_md.h}} under
> {{JAVA_HOME}}, rather than trying to look for them in system paths. Since we
> compile with Maven, we know that we'll have a valid {{JAVA_HOME}} at all
> times. There is no point digging in system paths, and it can lead to host
> contamination if the user has multiple JVMs installed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8765) LocalDirAllocator.ifExists API is broken and unused
Hemanth Yamijala created HADOOP-8765: Summary: LocalDirAllocator.ifExists API is broken and unused Key: HADOOP-8765 URL: https://issues.apache.org/jira/browse/HADOOP-8765 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Hemanth Yamijala Assignee: Hemanth Yamijala Priority: Minor LocalDirAllocator.ifExists calls AllocatorPerContext.ifExists, which accesses the localDirsPath variable that is uninitialised. Hence, any code that uses this API is likely to fail with a NullPointerException. However, this API is currently not used anywhere else in trunk. The earlier usage was in IsolationRunner, that has since been removed via MAPREDUCE-2606. Hence, we could remove this API for trunk, and if required, fix it for the 1.x branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448471#comment-13448471
]
Hudson commented on HADOOP-8764:
Integrated in Hadoop-Hdfs-trunk-Commit #2745 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2745/])
HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor
Robinson (Revision 1380984)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380984
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8737) cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
[
https://issues.apache.org/jira/browse/HADOOP-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448472#comment-13448472
]
Hudson commented on HADOOP-8737:
Integrated in Hadoop-Hdfs-trunk-Commit #2745 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2745/])
HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor
Robinson (Revision 1380984)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380984
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
> cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
> --
>
> Key: HADOOP-8737
> URL: https://issues.apache.org/jira/browse/HADOOP-8737
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8737.001.patch, HADOOP-8737.002.patch
>
>
> We should always use the {{libjvm.so}}, {{jni.h}}, and {{jni_md.h}} under
> {{JAVA_HOME}}, rather than trying to look for them in system paths. Since we
> compile with Maven, we know that we'll have a valid {{JAVA_HOME}} at all
> times. There is no point digging in system paths, and it can lead to host
> contamination if the user has multiple JVMs installed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8749) HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.
[ https://issues.apache.org/jira/browse/HADOOP-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448482#comment-13448482 ] Todd Lipcon commented on HADOOP-8749: - +1, looks good to me, please submit patch so that Hudson can check it out. > HADOOP-8031 changed the way in which relative xincludes are handled in > Configuration. > - > > Key: HADOOP-8749 > URL: https://issues.apache.org/jira/browse/HADOOP-8749 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: HADOOP-8749.patch, HADOOP-8749_rev2.patch > > > The patch from HADOOP-8031 changed the xml parsing to use > DocumentBuilder#parse(InputStream uri.openStream()) instead of > DocumentBuilder#parse(String uri.toString()).I looked into the implementation > of javax.xml.parsers.DocumentBuilder and org.xml.sax.InputSource and there is > a difference when the DocumentBuilder parse(String) method is used versus > parse(InputStream). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8765) LocalDirAllocator.ifExists API is broken and unused
[ https://issues.apache.org/jira/browse/HADOOP-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448487#comment-13448487 ] Hemanth Yamijala commented on HADOOP-8765: -- BTW, all other calls to LocalDirAllocator first call AllocatorPerContext.confChanged, that is responsible for initialising the localDirsPath. Inserting that call fixes the NullPointerException. > LocalDirAllocator.ifExists API is broken and unused > --- > > Key: HADOOP-8765 > URL: https://issues.apache.org/jira/browse/HADOOP-8765 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala >Priority: Minor > > LocalDirAllocator.ifExists calls AllocatorPerContext.ifExists, which accesses > the localDirsPath variable that is uninitialised. Hence, any code that uses > this API is likely to fail with a NullPointerException. > However, this API is currently not used anywhere else in trunk. The earlier > usage was in IsolationRunner, that has since been removed via MAPREDUCE-2606. > Hence, we could remove this API for trunk, and if required, fix it for the > 1.x branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8764) CMake: HADOOP-8737 broke ARM build
[
https://issues.apache.org/jira/browse/HADOOP-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448498#comment-13448498
]
Hudson commented on HADOOP-8764:
Integrated in Hadoop-Mapreduce-trunk-Commit #2706 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2706/])
HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor
Robinson (Revision 1380984)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380984
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
> CMake: HADOOP-8737 broke ARM build
> --
>
> Key: HADOOP-8764
> URL: https://issues.apache.org/jira/browse/HADOOP-8764
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.2.0-alpha
> Environment: Apache Maven 3.0.4
> Maven home: /usr/share/maven
> Java version: 1.7.0_06, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_06/jre
> Default locale: en_US, platform encoding: ISO-8859-1
> OS name: "linux", version: "3.2.0-1000-highbank", arch: "arm", family: "unix"
>Reporter: Trevor Robinson
>Assignee: Trevor Robinson
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8764.patch
>
>
> ARM build is broken again: CMAKE_SYSTEM_PROCESSOR comes from {{uname -p}},
> which reports values like "armv7l" for the ARMv7 architecture. However, the
> OpenJDK and Oracle ARM JREs both use "jre/lib/arm" for the JVM directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8737) cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
[
https://issues.apache.org/jira/browse/HADOOP-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448499#comment-13448499
]
Hudson commented on HADOOP-8737:
Integrated in Hadoop-Mapreduce-trunk-Commit #2706 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2706/])
HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor
Robinson (Revision 1380984)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380984
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
> cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
> --
>
> Key: HADOOP-8737
> URL: https://issues.apache.org/jira/browse/HADOOP-8737
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HADOOP-8737.001.patch, HADOOP-8737.002.patch
>
>
> We should always use the {{libjvm.so}}, {{jni.h}}, and {{jni_md.h}} under
> {{JAVA_HOME}}, rather than trying to look for them in system paths. Since we
> compile with Maven, we know that we'll have a valid {{JAVA_HOME}} at all
> times. There is no point digging in system paths, and it can lead to host
> contamination if the user has multiple JVMs installed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8749) HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.
[ https://issues.apache.org/jira/browse/HADOOP-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated HADOOP-8749: - Status: Patch Available (was: Open) > HADOOP-8031 changed the way in which relative xincludes are handled in > Configuration. > - > > Key: HADOOP-8749 > URL: https://issues.apache.org/jira/browse/HADOOP-8749 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: HADOOP-8749.patch, HADOOP-8749_rev2.patch > > > The patch from HADOOP-8031 changed the xml parsing to use > DocumentBuilder#parse(InputStream uri.openStream()) instead of > DocumentBuilder#parse(String uri.toString()).I looked into the implementation > of javax.xml.parsers.DocumentBuilder and org.xml.sax.InputSource and there is > a difference when the DocumentBuilder parse(String) method is used versus > parse(InputStream). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8766) FileContextMainOperationsBaseTest should randomize the root dir
Eli Collins created HADOOP-8766:
---
Summary: FileContextMainOperationsBaseTest should randomize the
root dir
Key: HADOOP-8766
URL: https://issues.apache.org/jira/browse/HADOOP-8766
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
FileContextMainOperationsBaseTest should randomize the name of the root
directory it creates. It currently hardcodes LOCAL_FS_ROOT_URI to {{/tmp/test}}.
This causes the job to fail if it clashes with another jobs that also uses that
path. Eg
{noformat}
org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a
directory: file:/tmp/test
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:362)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:931)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2333)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703)
at
org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testWorkingDirectory(FileContextMainOperationsBaseTest.java:178)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8765) LocalDirAllocator.ifExists API is broken and unused
[ https://issues.apache.org/jira/browse/HADOOP-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated HADOOP-8765: - Attachment: HADOOP-8765.patch Attached patch removes the ifExists API and AllocatorPerContext.ifExists API as well. Could someone please review ? > LocalDirAllocator.ifExists API is broken and unused > --- > > Key: HADOOP-8765 > URL: https://issues.apache.org/jira/browse/HADOOP-8765 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala >Priority: Minor > Attachments: HADOOP-8765.patch > > > LocalDirAllocator.ifExists calls AllocatorPerContext.ifExists, which accesses > the localDirsPath variable that is uninitialised. Hence, any code that uses > this API is likely to fail with a NullPointerException. > However, this API is currently not used anywhere else in trunk. The earlier > usage was in IsolationRunner, that has since been removed via MAPREDUCE-2606. > Hence, we could remove this API for trunk, and if required, fix it for the > 1.x branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8758) Support for pluggable token implementations
[ https://issues.apache.org/jira/browse/HADOOP-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448518#comment-13448518 ] Kan Zhang commented on HADOOP-8758: --- Thanks all for your comments. [~eric14], one use case is exactly the Gateway model you mentioned. Suppose a cluster can be accessed only through a gateway machine and the gateway is configured to authenticate its external clients using some other method (e.g., LDAP). We want to turn on Hadoop security within the cluster to support multi-tenancy, but we don't want to invest in Kerberos. One thing we can do is pre-generate and configure a shared key between Gateway and NN during install and use it to set up a secure connection between Gateway and NN in place of Kerberos. With pluggable interface to support multiple types of delegation tokens, such shared keys can be implemented in the form of a special kind of tokens and only connections authenticated using these special tokens can be used to fetch delegation tokens. In other words, these special tokens are used in place of Kerberos tickets to bootstrap security. The difference is Kerberos mechanism is more general; it can be used by any Kerberized clients without install changes. Whereas here we need to pre-configure our special tokens in a pair-wise manner for any pair of entities that we need to enable authenticated connections. Hence, its usefulness is limited to pre-known services, not arbitrary clients. [~revans2] and [~daryn], I agree we should allow "SIMPLE" auth to be coupled with tokens. [~owen.omalley], actually I'm proposing we use the same authentication mechanism as existing delegation tokens use (yes, we may need to consider upgrading the mechanism at some point and it depends on what's available in standard libs). What's new here is a new type of credentials, hence a new type of SecretManager to deal with it. Since the SecretManagers will be stacked together and deal with different types of credentials, they are orthogonal and can evolve independently. HTTP could be used for fetching tokens, but the root issue here is which auth method to use for intra-cluster authentication (ex. between Gateway and NN) and preferably we don't want to introduce an external dependency (ex, LDAP or Kerberos KDC). We could use a shared key based mechanism on HTTP, but that doesn't save us any trouble in configuring the shared keys. We might as well use RPC with its existing token framework. [~tucu00] Agree with decoupling intra-cluster authentication from user-facing authentication. This JIRA is a step in that direction. Regarding consolidating different token implementations, I see the benefit and have discussed it with you and others. Let's leave it to another topic. > Support for pluggable token implementations > --- > > Key: HADOOP-8758 > URL: https://issues.apache.org/jira/browse/HADOOP-8758 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, security >Reporter: Kan Zhang >Assignee: Kan Zhang > > Variants of the delegation token mechanism have been employed by different > Hadoop services (NN, JT, RM, etc) to re-authenticate a previously > Kerberos-authenticated client. While existing delegation token mechanism > compliments Kerberos well, it doesn't necessarily have to be coupled with > Kerberos. In principle, delegation tokens can be coupled with any > authentication mechanism that bootstraps security. In particular, it can be > coupled with other token implementations that use the same DIGEST-MD5 auth > method. For example, a token can be pre-generated in an out-of-band manner > and configured as a shared secret key between NN and JT to allow JT to make > initial authentication to NN. This simple example doesn't deal with token > renewal etc, but it helps to illustrate the point that if we can support > multiple pluggable token implementations, it opens up the possibility for > different users to plug in the token implementation of their choice to > bootstrap security. Such token based mechanism has advantages over Kerberos > in that 1) it doesn't require Kerberos infrastructure, 2) it leverages > existing SASL DIGEST-MD5 auth method and doesn't require adding a new RPC > auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8749) HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.
[ https://issues.apache.org/jira/browse/HADOOP-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448523#comment-13448523 ] Hadoop QA commented on HADOOP-8749: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543777/HADOOP-8749_rev2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.conf.TestConfiguration +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1400//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1400//console This message is automatically generated. > HADOOP-8031 changed the way in which relative xincludes are handled in > Configuration. > - > > Key: HADOOP-8749 > URL: https://issues.apache.org/jira/browse/HADOOP-8749 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: HADOOP-8749.patch, HADOOP-8749_rev2.patch > > > The patch from HADOOP-8031 changed the xml parsing to use > DocumentBuilder#parse(InputStream uri.openStream()) instead of > DocumentBuilder#parse(String uri.toString()).I looked into the implementation > of javax.xml.parsers.DocumentBuilder and org.xml.sax.InputSource and there is > a difference when the DocumentBuilder parse(String) method is used versus > parse(InputStream). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
