[jira] [Updated] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-10919: -- Attachment: HDFS-6134-Distcp-cp-UseCasesTable.pdf I have attached a table that shows the distcp/cp use cases and the desirable outcomes. I think this implementable in a transparent fashion within distcp or cp using /r/r mechanism. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch, > HDFS-6134-Distcp-cp-UseCasesTable.pdf > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096596#comment-14096596 ] Allen Wittenauer edited comment on HADOOP-9902 at 8/14/14 5:37 AM: --- Thanks [~rvs]! bq. It could have been done in multiple increments in a separate jiras to help review better. Not really. As has been pointed out before, once you touch hadoop-config.sh or *-env.sh in any sort of major way, you are pretty much touching everything since all the pieces are so tightly interlocked. As a result, you'd be reviewing the entire code base almost every time. Additionally, the whole point of my posting of test code, changes, random discussion points, etc as I went along was so that the 30+ people who have been watching this JIRA for almost a year now could point out dumb things I did and make suggestions. Some took advantage of it and helped me get rid of some stupidity on my part, either here in the JIRA or elsewhere. I owe much gratitude to them. :D It's probably worth pointing out that a good chunk of the new features have been floating around in JIRAs in patch available status since pre-v1 days. We clearly never cared enough to review these features when they were already separate and the patches viable. This was an opportunity to bring these good ideas and operational fixes forward. bq. Is there any concerns you see with the existing environment in mandating bash v3? Nope. Bash v3 shipped with, for example, Fedora 3 and Red Hat 8.x. These are operating systems that existed before Hadoop did. FreeBSD and NetBSD didn't, and may still not, ship bash at all as part of their core distribution. (It is, of course, in pkgsrc.) So we've been broken on them forever anyway. (Hai FreeBSD people who beat me up at conferences year after year... ) That release note is specifically for them since they always have to install bash anyway. I suppose we could always move to something like zsh which has a BSD-compatible license and then ship it with Hadoop too. (POSIX sh, BTW, is a significant performance drop.) (Some of the bash folks were completely surprised I made the requirement *so low* given that all modern OSes ship with v4.) bq. My concern is about the possible incompatibilities and breaking existing set of tools. Again, as discussed before, this is exactly why this is going into trunk and not branch-2. I'm treating this as an incompatible change even though I suspect that the vast majority of stuff will "just work". This comes from having used a form of this code for over a year now, both secure and insecure, multiple operating systems, multiple configs, multiple types of different ways to config, Hadoop v2.0.x through trunk, single hosts and multiple hosts, talking about config with folks at conferences, running through shellcheck, etc, etc. To me, the biggest, most potentially breaking change is really going to be the dropping of append in -env.sh. We've only gotten away with it because we've depended upon undocumented JVM behavior. But we can't dedupe JVM flags and support append in any sort of reliable manner. Given the number of complaints, questions, and even JIRAs around "why so many Xmx's?", it's clear that append had to go. But to restate, yet again, this is going into trunk. Stuff may break. Hopefully not, but if we can't put incompatible changes there, we've got bigger problems. was (Author: aw): Thanks [~rvs]! bq. It could have been done in multiple increments in a separate jiras to help review better. Not really. As has been pointed out before, once you touch hadoop-config.sh or *-env.sh in any sort of major way, you are pretty much touching everything since all the pieces are so tightly interlocked. As a result, you'd be reviewing the entire code base almost every time. Additionally, the whole point of my posting of test code, changes, random discussion points, etc as I went along was so that the 30+ people who have been watching this JIRA for almost a year now could point out dumb things I did and make suggestions. Some took advantage of it and helped me get rid of some stupidity on my part, either here in the JIRA or elsewhere. I owe much gratitude to them. :D It's probably worth pointing out that a good chunk of the new features have been floating around in JIRAs in patch available status since pre-v1 days. We clearly never cared enough to review these features when they were already separate and the patches viable. This was an opportunity to bring these good ideas and operational fixes forward. bq. Is there any concerns you see with the existing environment in mandating bash v3? Nope. Bash v3 shipped with, for example, Fedora 3 and Red Hat 8.x. These are operating systems that existed before Hadoop did. FreeBSD and NetBSD didn't, and may still not, ship bash at
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096596#comment-14096596 ] Allen Wittenauer commented on HADOOP-9902: -- Thanks [~rvs]! bq. It could have been done in multiple increments in a separate jiras to help review better. Not really. As has been pointed out before, once you touch hadoop-config.sh or *-env.sh in any sort of major way, you are pretty much touching everything since all the pieces are so tightly interlocked. As a result, you'd be reviewing the entire code base almost every time. Additionally, the whole point of my posting of test code, changes, random discussion points, etc as I went along was so that the 30+ people who have been watching this JIRA for almost a year now could point out dumb things I did and make suggestions. Some took advantage of it and helped me get rid of some stupidity on my part, either here in the JIRA or elsewhere. I owe much gratitude to them. :D It's probably worth pointing out that a good chunk of the new features have been floating around in JIRAs in patch available status since pre-v1 days. We clearly never cared enough to review these features when they were already separate and the patches viable. This was an opportunity to bring these good ideas and operational fixes forward. bq. Is there any concerns you see with the existing environment in mandating bash v3? Nope. Bash v3 shipped with, for example, Fedora 3 and Red Hat 8.x. These are operating systems that existed before Hadoop did. FreeBSD and NetBSD didn't, and may still not, ship bash at all as part of their core distribution. (It is, of course, in pkgsrc.) So we've been broken on them forever anyway. (Hai FreeBSD people who beat me up at conferences year after year... ) That release note is specifically for them since they always have to install bash anyway. I suppose we could always move to something like zsh which has a BSD-compatible license and then ship it with Hadoop too. (POSIX sh, BTW, is a significant performance drop.) (Some of the bash folks were completely surprised I made the requirement *so low* given that all modern OSes ship with v4.) bq. My concern is about the possible incompatibilities and breaking existing set of tools. Again, as discussed before, this is exactly why this is going into trunk and not branch-2. I'm treating this as an incompatible change even though I suspect that the vast majority of stuff will "just work". This comes from having used a form of this code for over a year now, both secure and insecure, multiple operating systems, multiple configs, multiple types of different ways to config, Hadoop v2.0.x through trunk, single hosts and multiple hosts, talking about config with folks at conferences, running through shellcheck, etc, etc. To me, the biggest, most potentially breaking change is really going to be the dropping of append in *-env.sh. We've only gotten away with it because we've depended upon undocumented JVM behavior. But we can't dedupe JVM flags *and* support append in any sort of reliable manner. Given the number of complaints, questions, and even JIRAs around "why so many Xmx's?", it's clear that append had to go. But to restate, yet again, this is going into trunk. Stuff may break. Hopefully not, but if we can't put incompatible changes there, we've got bigger problems. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10838) Byte array native checksumming
[ https://issues.apache.org/jira/browse/HADOOP-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096568#comment-14096568 ] Hudson commented on HADOOP-10838: - FAILURE: Integrated in Hadoop-trunk-Commit #6063 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6063/]) HADOOP-10838. Byte array native checksumming. Contributed by James Thomas. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617875) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCrc32.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c > Byte array native checksumming > -- > > Key: HADOOP-10838 > URL: https://issues.apache.org/jira/browse/HADOOP-10838 > Project: Hadoop Common > Issue Type: Improvement > Components: performance >Reporter: James Thomas >Assignee: James Thomas > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-10838.2.patch, HADOOP-10838.3.patch, > HADOOP-10838.4.patch, HADOOP-10838.patch, HDFS-3528.patch, HDFS-6560.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10838) Byte array native checksumming
[ https://issues.apache.org/jira/browse/HADOOP-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-10838: - Resolution: Fixed Fix Version/s: 2.6.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Byte array native checksumming > -- > > Key: HADOOP-10838 > URL: https://issues.apache.org/jira/browse/HADOOP-10838 > Project: Hadoop Common > Issue Type: Improvement > Components: performance >Reporter: James Thomas >Assignee: James Thomas > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-10838.2.patch, HADOOP-10838.3.patch, > HADOOP-10838.4.patch, HADOOP-10838.patch, HDFS-3528.patch, HDFS-6560.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10838) Byte array native checksumming
[ https://issues.apache.org/jira/browse/HADOOP-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096546#comment-14096546 ] Todd Lipcon commented on HADOOP-10838: -- +1, looks good. Will commit to trunk and branch-2 momentarily. Thanks James! > Byte array native checksumming > -- > > Key: HADOOP-10838 > URL: https://issues.apache.org/jira/browse/HADOOP-10838 > Project: Hadoop Common > Issue Type: Improvement > Components: performance >Reporter: James Thomas >Assignee: James Thomas > Attachments: HADOOP-10838.2.patch, HADOOP-10838.3.patch, > HADOOP-10838.4.patch, HADOOP-10838.patch, HDFS-3528.patch, HDFS-6560.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10838) Byte array native checksumming
[
https://issues.apache.org/jira/browse/HADOOP-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096527#comment-14096527
]
Hadoop QA commented on HADOOP-10838:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661617/HADOOP-10838.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4470//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4470//console
This message is automatically generated.
> Byte array native checksumming
> --
>
> Key: HADOOP-10838
> URL: https://issues.apache.org/jira/browse/HADOOP-10838
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance
>Reporter: James Thomas
>Assignee: James Thomas
> Attachments: HADOOP-10838.2.patch, HADOOP-10838.3.patch,
> HADOOP-10838.4.patch, HADOOP-10838.patch, HDFS-3528.patch, HDFS-6560.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10838) Byte array native checksumming
[ https://issues.apache.org/jira/browse/HADOOP-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HADOOP-10838: -- Attachment: HADOOP-10838.4.patch Thanks for the review, Todd. Updated patch. > Byte array native checksumming > -- > > Key: HADOOP-10838 > URL: https://issues.apache.org/jira/browse/HADOOP-10838 > Project: Hadoop Common > Issue Type: Improvement > Components: performance >Reporter: James Thomas >Assignee: James Thomas > Attachments: HADOOP-10838.2.patch, HADOOP-10838.3.patch, > HADOOP-10838.4.patch, HADOOP-10838.patch, HDFS-3528.patch, HDFS-6560.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10843) TestGridmixRecord unit tests failure on PowerPC
[
https://issues.apache.org/jira/browse/HADOOP-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096426#comment-14096426
]
Hudson commented on HADOOP-10843:
-
FAILURE: Integrated in Hadoop-trunk-Commit #6062 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6062/])
HADOOP-10843. TestGridmixRecord unit tests failure on PowerPC (Jinghui Wang via
Colin Patrick McCabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617860)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixRecord.java
> TestGridmixRecord unit tests failure on PowerPC
> ---
>
> Key: HADOOP-10843
> URL: https://issues.apache.org/jira/browse/HADOOP-10843
> Project: Hadoop Common
> Issue Type: Bug
> Components: test, tools
>Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.4.1
>Reporter: Jinghui Wang
>Assignee: Jinghui Wang
> Fix For: 2.6.0
>
> Attachments: HADOOP-10843.1.patch, HADOOP-10843.2.patch,
> HADOOP-10843.3.patch
>
>
> In TestGridmixRecord#binSortTest, the test expects the comparison result of
> WritableComparator.compareBytes, which uses UnsafeComparer, to be the integer
> difference rather than the documented "@return 0 if equal, < 0 if left is
> less than right, etc.".
> TestGridmixRecord#binSortTest code snippet
> {code}
> final int chk = WritableComparator.compareBytes(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength());
> assertEquals(chk, x.compareTo(y));
> assertEquals(chk, cmp.compare(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength()));
> {code}
> The code snippet below shows the Unsafe comparator behavior for
> non-little-endian machines.
> {code}
> if (!littleEndian) {
> return lessThanUnsigned(lw, rw) ? -1 : 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096420#comment-14096420 ] Andrew Wang commented on HADOOP-10919: -- Note that if you copy from at or above the EZ root, it'll preserve the EZ root's raw xattrs and thus create the EZ. We have a special hook in FSDirectory#unprotectedSetXAttrs that watches for the special EZ xattr being set. If you're copying from below the EZ root, then only that subtree is preserved. We don't automatically create an EZ above the distcp dst (which would be kind of weird behavior). > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10843) TestGridmixRecord unit tests failure on PowerPC
[
https://issues.apache.org/jira/browse/HADOOP-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-10843:
--
Resolution: Fixed
Fix Version/s: 2.6.0
Target Version/s: 2.6.0
Status: Resolved (was: Patch Available)
> TestGridmixRecord unit tests failure on PowerPC
> ---
>
> Key: HADOOP-10843
> URL: https://issues.apache.org/jira/browse/HADOOP-10843
> Project: Hadoop Common
> Issue Type: Bug
> Components: test, tools
>Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.4.1
>Reporter: Jinghui Wang
>Assignee: Jinghui Wang
> Fix For: 2.6.0
>
> Attachments: HADOOP-10843.1.patch, HADOOP-10843.2.patch,
> HADOOP-10843.3.patch
>
>
> In TestGridmixRecord#binSortTest, the test expects the comparison result of
> WritableComparator.compareBytes, which uses UnsafeComparer, to be the integer
> difference rather than the documented "@return 0 if equal, < 0 if left is
> less than right, etc.".
> TestGridmixRecord#binSortTest code snippet
> {code}
> final int chk = WritableComparator.compareBytes(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength());
> assertEquals(chk, x.compareTo(y));
> assertEquals(chk, cmp.compare(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength()));
> {code}
> The code snippet below shows the Unsafe comparator behavior for
> non-little-endian machines.
> {code}
> if (!littleEndian) {
> return lessThanUnsigned(lw, rw) ? -1 : 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096400#comment-14096400 ] Roman Shaposhnik commented on HADOOP-9902: -- Just to close a loop on this from my side: at this point I'm confident in this change to go into trunk. Sure we may discover small hicups, but the bulk of it is extremely solid. +1 > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10959) A Complement and Short Term Solution to TokenAuth Based on Kerberos Pre-Authentication Framework
[ https://issues.apache.org/jira/browse/HADOOP-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096392#comment-14096392 ] Kai Zheng commented on HADOOP-10959: bq.I'm not sure why generating principals and keytabs seems to be viewed as a difficult activity. Yes generating principals and keytabs shouldn't be difficult in normal form, but would be difficult in a dynamic scheduled environment like YARN. We should also consider the distribution of the keytabs. Regarding this, perhaps developers from Slider project have the real feeling. bq.Most if not nearly all large corporations already use kerberos in some form Quite agree, so this solution bases on Kerberos to support the token mechanism. We're not coming up something in line with Kerberos or the Kerberos Authentication Method in Hadoop, but enhance and extend them compatibly. bq.My impression it's a misnomer to call this a pre-auth extension? It's not a hardening like the encrypted timestamp pre-auth, but instead is a complete substitute for a password/keytab. Doesn't this completely undermine kerberos? Well, let me update verbosely. Classically in RFC4120, Kerberos defined a preauth phase in KDC exchange to alleviate password attack, your mentioned timestamp preauth is right here. Note by employing the preauth, it also allowed to extend the Kerberos protocol and prove the identity of a principal otherwise than just password. In fact PKINIT(RFC4556) was defined to prove principal identity with x509 certificate. Then, FAST(RFC6113) defines a a generalized framework for better defining a preauth mechanism, and in this framework OTP(RFC6560) was defined, and token-preauth is being defined. OTP preauth allows to use a one time password, and token-preauth use a JWT token, to prove the identity instead of password. For end users (not service principal), in fact password or key won't be a must and can be purged at all. Note both AD and MIT Kerberos implemented FAST and PKINIT, so don't worry about it. token-preauth conforms to all these and won't undermine Kerberos. bq.Will cross-realm authentication work? Do all KDCs in the trust have to use the custom plugin? Cross-realm trust works in Kerberos itself, and in practice MIT KDC trusting AD is deployed well. Like trusting another KDC, a KDC can trust a token issuer/authority. All the trusts can work together or by itself, but not get mixed together. So a MIT KDC can trust an AD for a set of users and a token authority for another set of users at the same time in a deployment, and the token-preauth plugin is only needed by the MIT KDC. bq.how do tasks running in the cluster authenticate under the model? Do they continue to use the existing delegation tokens obtained via JWT/TGT during job submission I would think yes. The JWT token support is mainly for the initial authentication before accessing to Hadoop. I think delegation token works well internally to bypass some Kerberos constraint. Before we could push Kerberos to evolve to resolve such limits, nothing we can do for now. bq.are they using the JWT tokens to obtain a TGT/TGS in the tasks? I think the latter? Great idea. Should this go so far I don't know. Maybe in the long future Hadoop can achieve to that, all tokens are unified into one (a standardized JWT token that works both internally and externally), but not for now I guess. bq.Mention is made of an AD-TOKEN which I believe is a non-standard MS extension? Do you envision the JWT issuer containing the group mapping for all services? Yes token-preauth (with the new AD-TOKEN) is to be standardized. In this extension, JWT token MAY(not REQUIRE) contain groups and also other useful attributes. If it does, then such attribute(s) can be extracted and employed for authorization. If otherwise, no hurt at all and groups mapping can still fall back to original providers. bq.something like certificates or private keys or shared secret to be involved - which would be an equal but different "pain to deploy"? Yes certificates or tokens may be involved, but can be done with less (not just different) pain. In YARN/Slider totally dynamically scheduled environment, think about how to prepare for the long time running service principals and keytabs to be scheduled to run in dynamic containers? > A Complement and Short Term Solution to TokenAuth Based on Kerberos > Pre-Authentication Framework > > > Key: HADOOP-10959 > URL: https://issues.apache.org/jira/browse/HADOOP-10959 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: Rhino > Attachments: KerbToken-v2.pdf > > > To implement and integrate pluggable authenti
[jira] [Comment Edited] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096244#comment-14096244 ] Suresh Srinivas edited comment on HADOOP-9902 at 8/14/14 1:03 AM: -- [~chris.douglas], this is very good work by [~aw] to do the much needed cleanup. However other than Roman, I have not seen any committer review this change thoroughly and ready to +1 it. Even Roman has a bunch of caveats. Not sure if reviews can be effective where rewrite and addition of functionality all has happened together. If the only concern is this patch becoming stale, I will help in rebasing it. was (Author: sureshms): [~chris.douglas], this is very good work done by [~aw] to do the much needed cleanup. However other than Roman, I have not seen any committer review this change thoroughly and ready to +1 it. Even Roman has bunch of caveats. Not sure if reviews can be effective where rewrite and addition of functionality all has happened together. It the only concern is this patch becoming stale, I will help in rebasing it. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-7713) dfs -count -q should label output column
[
https://issues.apache.org/jira/browse/HADOOP-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096371#comment-14096371
]
Allen Wittenauer commented on HADOOP-7713:
--
To me, count has always been in line with the Unix df, quota, iostat, zfs, etc
storage commands where it's just default behavior to print column names. (du
and ls are practically outliers!) The fact that it doesn't, I always marked up
as naivety in how the Unix-equivalent commands worked. So while I can
appreciate that it would be useful to have in 2.x, I just think long term it's
the wrong behavior not to put the columns up by default. Casual users will
still be annoyed/confused if we don't. Plus, as you said, it's a nice to have,
not a necessity in 2.x.
I definitely agree about the column names. Spaces = bad.
> dfs -count -q should label output column
>
>
> Key: HADOOP-7713
> URL: https://issues.apache.org/jira/browse/HADOOP-7713
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nigel Daley
>Assignee: Jonathan Allen
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
> HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch
>
>
> These commands should label the output columns:
> {code}
> hadoop dfs -count ...
> hadoop dfs -count -q ...
> {code}
> Current output of the 2nd command above:
> {code}
> % hadoop dfs -count -q /user/foo /tmp
> none inf 9569 9493 6372553322
> hdfs://nn1.bar.com/user/foo
> none inf 101 2689 209349812906
> hdfs://nn1.bar.com/tmp
> {code}
> It is not obvious what these columns mean.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-7713) dfs -count -q should label output column
[
https://issues.apache.org/jira/browse/HADOOP-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-7713:
-
Status: Open (was: Patch Available)
> dfs -count -q should label output column
>
>
> Key: HADOOP-7713
> URL: https://issues.apache.org/jira/browse/HADOOP-7713
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nigel Daley
>Assignee: Jonathan Allen
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
> HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch
>
>
> These commands should label the output columns:
> {code}
> hadoop dfs -count ...
> hadoop dfs -count -q ...
> {code}
> Current output of the 2nd command above:
> {code}
> % hadoop dfs -count -q /user/foo /tmp
> none inf 9569 9493 6372553322
> hdfs://nn1.bar.com/user/foo
> none inf 101 2689 209349812906
> hdfs://nn1.bar.com/tmp
> {code}
> It is not obvious what these columns mean.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096358#comment-14096358 ] Charles Lamb commented on HADOOP-10919: --- bq. Q. when you say "distcp /r/r/src /r/r/dest" are the keys read from src and preserved in the dest? Does the act of copying the keys from a /r/r/src into a /r/r/dest automatically set up a matching EZ in the destination? Yes to the first question and no to the second. Copying the keys occurs and that is almost good enough to set up a matching EZ. However, what doesn't happen is a call to createEncryptionZone so there is not an actual EZ in place on the dst. The admin is expected to have done that before the distcp. If the admin wants a parallel EZ (i.e. with the same keys, ez-key, etc.) -- and presumably they do because they're copying from /.r/r to /.r/r and preserving the keys along the way (this is my case "(a)" above) -- then it is also expected that if the dest NN is not the same as the src (likely) that the NN and the clients accessing that NN will have equal access to the KMS (presumably the same KMS is shared across src and dst). Does this make sense? > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096353#comment-14096353 ] Sanjay Radia commented on HADOOP-10919: --- Q. when you say "distcp /r/r/src /r/r/dest" are the keys read from src and preserved in the dest? Does the act of copying the keys from a /r/r/src into a /r/r/dest automatically set up a matching EZ in the destination? > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096348#comment-14096348 ] Alejandro Abdelnur commented on HADOOP-10880: - daryn, i've thinking about this. if we do that, how non-java clients will do it? a curl client for example. > Move HTTP delegation tokens out of URL querystring to a header > -- > > Key: HADOOP-10880 > URL: https://issues.apache.org/jira/browse/HADOOP-10880 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Blocker > Attachments: HADOOP-10880.patch, HADOOP-10880.patch, > HADOOP-10880.patch > > > Following up on a discussion in HADOOP-10799. > Because URLs are often logged, delegation tokens may end up in LOG files > while they are still valid. > We should move the tokens to a header. > We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096334#comment-14096334 ] Daryn Sharp commented on HADOOP-10880: -- Is it reasonable to implement this with digest auth now rather than later? I think you've already got the hooks to do it. We've already got insecure token in the query string, this would be insecure token in a header, whereas digest will be secure token in a header on par with rpc. Taking away functionality once it's unleashed is hard and requires complicated backwards compatibility. When it comes to authentication code, more so than anywhere else, less is better. It's too easy to make subtle mistakes that compromise the system... > Move HTTP delegation tokens out of URL querystring to a header > -- > > Key: HADOOP-10880 > URL: https://issues.apache.org/jira/browse/HADOOP-10880 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Blocker > Attachments: HADOOP-10880.patch, HADOOP-10880.patch, > HADOOP-10880.patch > > > Following up on a discussion in HADOOP-10799. > Because URLs are often logged, delegation tokens may end up in LOG files > while they are still valid. > We should move the tokens to a header. > We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10893) isolated classloader on the client side
[
https://issues.apache.org/jira/browse/HADOOP-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096331#comment-14096331
]
Sangjin Lee commented on HADOOP-10893:
--
Thanks for the review Jason! It's helpful as always.
bq. HADOOP_USE_CLIENT_CLASSLOADER would be a better name to keep within the
HADOOP_* shell variable namespace and is consistent with
HADOOP_USER_CLASSPATH_FIRST. Similarly HADOOP_CLIENT_SYSTEM_CLASSES.
Sounds good. I'll make the change.
bq. ApplicationClassLoader was marked Public, so I'm wondering if we should
leave a deprecated, trivial derivation of the new class location just in case
someone referenced it?
I did not notice that it was marked public. I'll recreate a deprecated
extending class in its current location.
bq. What was the rationale behind the Splitter change which seems unrelated?
If possible, I wanted to avoid having a dependency from this classloader class
to another library unless it's really necessary. Splitter was coming from
guava. :) In theory it should be OK even if ApplicationClassLoader used a guava
class. It would be loaded by the system classloader anyway, and it would not
interfere with the ApplicationClassLoader's ability to load a new version of
the class for the user.
However, it was more of a call to minimize the external dependency from
ApplicationClassLoader. I believe the current version (using String.split()) is
equivalent and using the Splitter is not needed, but I'd be open to reversing
it.
{quote}
Would be nice if we could somehow tie the default system classes defined in
RunJar with the default for the job classloader so we don't have to remember to
change it in two places going forward. Unfortunately the job classloader one is
encoded in mapred-default.xml, so I don't know of a good way to do this
offhand. Any ideas?
{quote}
I struggled with that decision a bit. As you mentioned, if you want to override
the defaults, you'd need to do it in two places if you use it for the client
and for the tasks as well (and for the vast majority of the cases I would
imagine that is the case).
At least I feel that it would be better if at least the default is in one
place. In that sense, how about having the default in ApplicationClassLoader
itself? You still need to override it in two places, but it feels like an
improvement over the current version.
{quote}
The doc comments in hadoop-config.sh should mention the client system classes
variable, how to use it, and potentially even its default value. I know, I
know. Yet another place to update if it changes, but users will likely have
easy access to the config comment and not the java/javadoc for RunJar. Or maybe
the default should already be in hadoop-config.sh with a hardcoded, last-resort
fallback in RunJar if not set in hadoop-config.sh? Anyway we should at least
mention the ability to specify the system classes.
{quote}
I agree it would be good to document the usage of the system classes env
variable. I'll add the comment to hadoop-config.sh. See above for where to
define the default and let me know what you think.
{quote}
Would be nice if we could have a unit test to verify the functionality is
working going forward. Maybe a unit test that writes out some app code in a
jar, has RunJar run it with the client classloader, and the app code verifies
it has appropriate classpath semantics? Thinking something along the lines of
how TestApplicationClassloader works but verifying RunJar setup the
classloaders properly.
{quote}
Let me look into a unit test for this that involves RunJar. Do you happen to
know of an existing test that writes out classes/jars off the top of your head?
{quote}
Nit: Not thrilled to see that the variable just has to be defined to anything,
although I see HADOOP_USER_CLASSPATH_FIRST set a precedent for it. Leads to
unexpected behavior if a user sees something like
HADOOP_USER_CLASSPATH_FIRST=true and tries HADOOP_USER_CLASSPATH_FIRST=false.
Not a must-fix, but it'd be nice to only accept expected values for the
variable. A shell func to sanity-check a boolean env would be helpful, maybe
something to tackle in a followup JIRA.
{quote}
Yes I stumbled on that as well, and it struck me as an odd behavior. I think
I'll file a separate JIRA to tackle that issue...
> isolated classloader on the client side
> ---
>
> Key: HADOOP-10893
> URL: https://issues.apache.org/jira/browse/HADOOP-10893
> Project: Hadoop Common
> Issue Type: New Feature
> Components: util
>Affects Versions: 2.4.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: HADOOP-10893.patch, HADOOP-10893.patch,
> classloader-test.tar.gz
>
>
> We have the job classloader on the mapreduce tasks that run on the cluster.
> It has a benefit of being abl
[jira] [Commented] (HADOOP-10893) isolated classloader on the client side
[ https://issues.apache.org/jira/browse/HADOOP-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096280#comment-14096280 ] Jason Lowe commented on HADOOP-10893: - Thanks, Sangjin! Some comments on the patch: HADOOP_USE_CLIENT_CLASSLOADER would be a better name to keep within the HADOOP_* shell variable namespace and is consistent with HADOOP_USER_CLASSPATH_FIRST. Similarly HADOOP_CLIENT_SYSTEM_CLASSES. ApplicationClassLoader was marked Public, so I'm wondering if we should leave a deprecated, trivial derivation of the new class location just in case someone referenced it? What was the rationale behind the Splitter change which seems unrelated? Would be nice if we could somehow tie the default system classes defined in RunJar with the default for the job classloader so we don't have to remember to change it in two places going forward. Unfortunately the job classloader one is encoded in mapred-default.xml, so I don't know of a good way to do this offhand. Any ideas? The doc comments in hadoop-config.sh should mention the client system classes variable, how to use it, and potentially even its default value. I know, I know. Yet another place to update if it changes, but users will likely have easy access to the config comment and not the java/javadoc for RunJar. Or maybe the default should already be in hadoop-config.sh with a hardcoded, last-resort fallback in RunJar if not set in hadoop-config.sh? Anyway we should at least mention the ability to specify the system classes. Would be nice if we could have a unit test to verify the functionality is working going forward. Maybe a unit test that writes out some app code in a jar, has RunJar run it with the client classloader, and the app code verifies it has appropriate classpath semantics? Thinking something along the lines of how TestApplicationClassloader works but verifying RunJar setup the classloaders properly. Nit: Not thrilled to see that the variable just has to be defined to anything, although I see HADOOP_USER_CLASSPATH_FIRST set a precedent for it. Leads to unexpected behavior if a user sees something like HADOOP_USER_CLASSPATH_FIRST=true and tries HADOOP_USER_CLASSPATH_FIRST=false. Not a must-fix, but it'd be nice to only accept expected values for the variable. A shell func to sanity-check a boolean env would be helpful, maybe something to tackle in a followup JIRA. > isolated classloader on the client side > --- > > Key: HADOOP-10893 > URL: https://issues.apache.org/jira/browse/HADOOP-10893 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Affects Versions: 2.4.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: HADOOP-10893.patch, HADOOP-10893.patch, > classloader-test.tar.gz > > > We have the job classloader on the mapreduce tasks that run on the cluster. > It has a benefit of being able to isolate class space for user code and avoid > version clashes. > Although it occurs less often, version clashes do occur on the client JVM. It > would be good to introduce an isolated classloader on the client side as well > to address this. A natural point to introduce this may be through RunJar, as > that's how most of hadoop jobs are run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096249#comment-14096249 ] Suresh Srinivas commented on HADOOP-9902: - bq. While thorough review could take awhile, validating the general direction should be quick. I am happy with the general direction. My concern is about the possible incompatibilities and breaking existing set of tools. Also bugs (which we can always fix them and stabilize in trunk). > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096244#comment-14096244 ] Suresh Srinivas commented on HADOOP-9902: - [~chris.douglas], this is very good work done by [~aw] to do the much needed cleanup. However other than Roman, I have not seen any committer review this change thoroughly and ready to +1 it. Even Roman has bunch of caveats. Not sure if reviews can be effective where rewrite and addition of functionality all has happened together. It the only concern is this patch becoming stale, I will help in rebasing it. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096236#comment-14096236 ] Chris Douglas commented on HADOOP-9902: --- [~sureshms]: the current patch has received a fair amount of feedback and the testing will become stale. Could you complete a review this week? While thorough review could take awhile, validating the general direction should be quick. The details can be worked out as followup, if you're satisfied with the cleanup generally. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10770) KMS add delegation token support
[
https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096227#comment-14096227
]
Hadoop QA commented on HADOOP-10770:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661536/HADOOP-10770.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4469//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4469//console
This message is automatically generated.
> KMS add delegation token support
>
>
> Key: HADOOP-10770
> URL: https://issues.apache.org/jira/browse/HADOOP-10770
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10770.patch, HADOOP-10770.patch,
> HADOOP-10770.patch, HADOOP-10770.patch
>
>
> This is a follow up on HADOOP-10769 for KMS itself.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096204#comment-14096204 ] Suresh Srinivas commented on HADOOP-9902: - [~aw], I would like to review these scripts as well. Please give time till the next Wednesday (earlier if I can find time). In general this is a major rewrite. It could have been done in multiple increments in a separate jiras to help review better. Some high level comments - Is there any concerns you see with the existing environment in mandating bash v3? Also can you please add new functionality (jnipath, distch) to a separate jira, instead of mixing it with rewrite. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10957) The globber will sometimes erroneously return a permission denied exception when there is a non-terminal wildcard
[ https://issues.apache.org/jira/browse/HADOOP-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096186#comment-14096186 ] Daryn Sharp commented on HADOOP-10957: -- If you insist on splitting up the patch, take all the test changes on HADOOP-10942. The existing tests were sufficient if permissions are enabled and the the priv/unpriv fs are handled as intended. > The globber will sometimes erroneously return a permission denied exception > when there is a non-terminal wildcard > - > > Key: HADOOP-10957 > URL: https://issues.apache.org/jira/browse/HADOOP-10957 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10957.001.patch, HADOOP-10957.002.patch > > > The globber will sometimes erroneously return a permission denied exception > when there is a non-terminal wildcard. The existing unit tests don't catch > this, because it doesn't happen for superusers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10838) Byte array native checksumming
[
https://issues.apache.org/jira/browse/HADOOP-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096183#comment-14096183
]
Todd Lipcon commented on HADOOP-10838:
--
Patch looks good except for one small issue:
{code}
+if (unlikely(ret == INVALID_CHECKSUM_DETECTED)) {
+ long pos = base_pos + (error_data.bad_data - data) + checksumNum *
+bytes_per_checksum;
+ throw_checksum_exception(
+env, error_data.got_crc, error_data.expected_crc,
+j_filename, pos);
+} else if (unlikely(ret != CHECKSUMS_VALID)) {
+ THROW(env, "java/lang/AssertionError",
+"Bad response code from native bulk_verify_crc");
+}
{code}
In both of these error conditions, you need a 'return', or else you'll go back
through the loop again. That would likely cause a crash -- it's illegal to make
most JNI calls while an exception is pending.
+1 once that's addressed
> Byte array native checksumming
> --
>
> Key: HADOOP-10838
> URL: https://issues.apache.org/jira/browse/HADOOP-10838
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance
>Reporter: James Thomas
>Assignee: James Thomas
> Attachments: HADOOP-10838.2.patch, HADOOP-10838.3.patch,
> HADOOP-10838.patch, HDFS-3528.patch, HDFS-6560.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096167#comment-14096167 ] Daryn Sharp commented on HADOOP-8989: - Unless I missed them, add tests to ensure that relative paths are displayed as relative paths, and no path defaults to pwd (which is relative too) and I'm +1. > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-8944: - Assignee: Jonathan Allen (was: Allen Wittenauer) > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Jonathan Allen >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096162#comment-14096162 ] Daryn Sharp commented on HADOOP-8989: - It should be "hadoop fs -find". FsShell is not hdfs specific. We should have deprecated "hdfs dfs" long ago... All it effectively does is call "hadoop fs". Does it make sense to "hdfs dfs -ls file:/tmp"? > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10698: Attachment: HADOOP-10698.patch rebasing to trunk > KMS, add proxyuser support > -- > > Key: HADOOP-10698 > URL: https://issues.apache.org/jira/browse/HADOOP-10698 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10698.patch, HADOOP-10698.patch, > HADOOP-10698.patch, HADOOP-10698.patch > > > Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10880: Attachment: HADOOP-10880.patch rebasing to trunk. > Move HTTP delegation tokens out of URL querystring to a header > -- > > Key: HADOOP-10880 > URL: https://issues.apache.org/jira/browse/HADOOP-10880 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Blocker > Attachments: HADOOP-10880.patch, HADOOP-10880.patch, > HADOOP-10880.patch > > > Following up on a discussion in HADOOP-10799. > Because URLs are often logged, delegation tokens may end up in LOG files > while they are still valid. > We should move the tokens to a header. > We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096114#comment-14096114
]
Hudson commented on HADOOP-10966:
-
FAILURE: Integrated in Hadoop-trunk-Commit #6060 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6060/])
HADOOP-10966. Hadoop Common native compilation broken in windows. (Contributed
by David Villegas) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617823)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Status: Patch Available (was: Open) > KMS add delegation token support > > > Key: HADOOP-10770 > URL: https://issues.apache.org/jira/browse/HADOOP-10770 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10770.patch, HADOOP-10770.patch, > HADOOP-10770.patch, HADOOP-10770.patch > > > This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Attachment: HADOOP-10770.patch rebasing to trunk. > KMS add delegation token support > > > Key: HADOOP-10770 > URL: https://issues.apache.org/jira/browse/HADOOP-10770 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10770.patch, HADOOP-10770.patch, > HADOOP-10770.patch, HADOOP-10770.patch > > > This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Issue Type: Improvement (was: Bug) > Replace HttpFS custom proxyuser handling with common implementation > --- > > Key: HADOOP-10836 > URL: https://issues.apache.org/jira/browse/HADOOP-10836 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, > HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch > > > Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Issue Type: Bug (was: Sub-task) Parent: (was: HADOOP-10822) > Replace HttpFS custom proxyuser handling with common implementation > --- > > Key: HADOOP-10836 > URL: https://issues.apache.org/jira/browse/HADOOP-10836 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, > HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch > > > Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10822) Refactor HTTP proxyuser support out of HttpFS into common
[ https://issues.apache.org/jira/browse/HADOOP-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved HADOOP-10822. - Resolution: Invalid converted HADOOP-10835 and HADOOP-10836 to issues as HADOOP-10836 should be an HDFS issue and subtasks cannot belong to different projects. > Refactor HTTP proxyuser support out of HttpFS into common > - > > Key: HADOOP-10822 > URL: https://issues.apache.org/jira/browse/HADOOP-10822 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > > HttpFS implements HTTP proxyuser support inline in httpfs code. > For HADOOP-10698 we need similar functionality for KMS. > Not to duplicate code, we should refactor existing code to common. > We should also leverage HADOOP-10817. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10835:
Issue Type: Improvement (was: Sub-task)
Parent: (was: HADOOP-10822)
> Implement HTTP proxyuser support in HTTP authentication client/server
> libraries
> ---
>
> Key: HADOOP-10835
> URL: https://issues.apache.org/jira/browse/HADOOP-10835
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 2.6.0
>
> Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
> HADOOP-10835.patch, HADOOP-10835.patch
>
>
> This is to implement generic handling of proxyuser in the
> {{DelegationTokenAuthenticatedURL}} and
> {{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
> the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096041#comment-14096041
]
David Villegas commented on HADOOP-10966:
-
Thanks both!
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[
https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096037#comment-14096037
]
Charles Lamb commented on HADOOP-10919:
---
I'll update the HDFS-6509 doc to reflect the bit about trashing.
{quote}
1.src subtree and dst subtree do not have EZ - easy, same as today
{quote}
Agreed.
{quote}
2.src subtree has no EZ but dest does have EZ in a portion of its subtree.
Possible outcomes
1. if user performing operation has permissions in dest EZ then the files
within the dest EZ subtree are encrypted
{quote}
Agreed.
{quote}
2.src subtree has no EZ but dest does have EZ in a portion of its subtree.
Possible outcomes
...
2. if user does not (say Admin) what do we expect to happen?
{quote}
The behavior should be the same as what happens today: user (the admin) gets a
permission violation because the admin does not have access to the target.
{quote}
3.src subtree has EZ but dest does not. Possible outcomes
1. files copied as encrypted but cannot be decryptied at the dest since
it does not have an EZ zone- useful as a backup
{quote}
/.r/r: raw files are copied to dest so dest contains encrypted (and unreadable)
files
!/.r/r: files are decrypted by distcp and copied to dst (decrypted). Files are
readable because they have been decrypted during the copy.
{quote}
3.src subtree has EZ but dest does not. Possible outcomes
...
2. files copied as encrypted and a matching EZ is created automatically.
Can an admin do this operation since he does not have access to the keys?
{quote}
I don't think that distcp can, or should, create a matching EZ automatically.
It is too hard for it to know what the intent of the copy is. Should the new ez
have the same ez-key as the src ez or a different one? Sure, we could have an
option to let the user specify that, but for the first crack I wanted to keep
it fairly simple. So, the theory is that the admin creates the empty EZ before
performing the distcp. The admin can either set up the EZ with the same ez-key
as the src ez (call this "(a)" below, or the dest can have a different ez-key
than the src (call this "(b)" below. After the ez is created, then distcp will
try to maintain the files as encrypted. In either of those scenarios, there are
a couple of cases:
distcp with /.r/r: (a) works ok because the EDEKs for each file are copied from
src to dst. (b) does not work because when the files are opened in the dest
hierarchy, the EDEKs will be decrypted with the new ez-key(dst) and that won't
work. This could be made to work by having the KMS decrypt the EDEKs and
re-encrypt them with the new ez-key(dst), but it would assume that the distcp
invoker had proper credentials with the KMS for the keys. So in general this
scenario is only useful when the src-ez and the dst-ez have been setup with the
same ez-key. There are other issues with this that are discussed under
HDFS-6134, such as different key lengths, etc.
distcp with no /.r/r: Both of (a) and (b) work ok as long as the invoker has
access to the files that are being copied. distcp decrypts the files on read
and they get re-encrypted on write. This is pretty much the same as today.
{quote}
3.src subtree has EZ but dest does not. Possible outcomes
...
3. throw an error which can be overidden by a flag in which case the
files are decryoted and copied to in dest are left decrypted . This only works
if the user has permissions for decryption; admin cannot do this.
{quote}
/.r/r: The files aren't decrypted so this scenario is perfectly acceptable.
!/.r/r: As you say, the admin can't do this because they presumably don't have
access to the files on the src (and probably not on the target either). So this
scenario is about some random user doing a distcp of some subset of the tree on
their own. I think that what you're suggesting is a way of trying to keep the
user from shooting themselves in the foot by ensuring that they don't leave
unencrypted data hanging around in the dest. I can see this both ways. On the
one hand, someone has given the user access to the files and keys. They are
expected to "do the right thing" with the decrypted file contents, including
not putting it somewhere "unsafe". It is "transparent encryption" after all.
And they might actually want to leave it hanging around in unencrypted form
because (e.g.) maybe dst is on a cluster inside a SCIF and it's ok to leave the
files unencrypted.
But I think I like your suggestion that we throw an exception in this case
(user not using /.r/r, any of the source paths are in an ez, dest is not in an
ez) unless a flag is set.
{quote}
4.both src and dest have EZ at exactly the same part of the subtree.
Possible outcomes
1. If user has permission to decrypt and encrypt, then the data is copied
and encryption is redone with new keys,
[jira] [Updated] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-10966:
---
Resolution: Fixed
Fix Version/s: 2.6.0
3.0.0
Target Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
No new tests required as it is a build break fix.
I committed it to trunk and branch-2. Thanks for the prompt fix [~dvillegas].
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096031#comment-14096031
]
Arpit Agarwal edited comment on HADOOP-10966 at 8/13/14 8:09 PM:
-
No new tests required as it is a build break fix.
I committed it to trunk and branch-2. Thanks for the prompt fix [~dvillegas]
and thanks [~vinayrpet] for the verification.
was (Author: arpitagarwal):
No new tests required as it is a build break fix.
I committed it to trunk and branch-2. Thanks for the prompt fix [~dvillegas].
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096032#comment-14096032 ] Aaron T. Myers commented on HADOOP-10836: - Sounds good. Thanks, Tucu. +1 > Replace HttpFS custom proxyuser handling with common implementation > --- > > Key: HADOOP-10836 > URL: https://issues.apache.org/jira/browse/HADOOP-10836 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, > HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch > > > Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Fix Version/s: 3.0.0 > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Fix For: 3.0.0 > > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Attachment: HADOOP-10836.patch [~atm], thxs for the review. yes, those 2 properties should be commented out, i've uploaded a new patch with the following comment at the beginning of the commented section: The following 2 properties within this comment are provided as an example to facilitate configuring HttpFS proxyusers. committing momentarily. > Replace HttpFS custom proxyuser handling with common implementation > --- > > Key: HADOOP-10836 > URL: https://issues.apache.org/jira/browse/HADOOP-10836 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, > HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch > > > Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-7099) Make RawLocalFileSystem more friendly to sub-classing
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatesh updated HADOOP-7099: -- Priority: Major (was: Minor) > Make RawLocalFileSystem more friendly to sub-classing > - > > Key: HADOOP-7099 > URL: https://issues.apache.org/jira/browse/HADOOP-7099 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 0.23.0 >Reporter: Noah Watkins > Attachments: HADOOP-7099.patch > > > This patch does 3 things that makes sub-classing RawLocalFileSystem easier. > First, it adds a constructor that allows a sub-class to avoid calling > getInitialWorkingDirectory(). This is important because if a sub-class has an > initially null uri (prior to initialize() being called), then > getInitialWorkingDirectory() will cause a NullPointerException when it tries > to work with the FS's uri. > Second, allows subclasses to modify the working directory. > The third thing this patch does is change loadPermissions to not pass the URI > object to the File(URI) constructor, but rather pass the string > representation of the path. This is important because URI's that are not > using the "file" scheme will cause the File(URI) constructor to throw an > exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10969) RawLocalFileSystem.setPermission throws Exception
Venkatesh created HADOOP-10969:
--
Summary: RawLocalFileSystem.setPermission throws Exception
Key: HADOOP-10969
URL: https://issues.apache.org/jira/browse/HADOOP-10969
Project: Hadoop Common
Issue Type: Bug
Environment: hadoop 2.3.0, Windows Environment, Development using
Eclipse, Lenevo Laptop
Reporter: Venkatesh
Priority: Blocker
I'm an application developer. We recently moved from CDH4.7 to CDH5.1. The
hadoop version have been from 1.x to 2.x. In order to perform development on
Eclipse (on WINDOWS), the following class was created
public class WindowsLocalFileSystem extends LocalFileSystem {
public WindowsLocalFileSystem() {
super();
}
@Override
public boolean mkdirs(Path f, FsPermission permission) throws
IOException {
final boolean result = super.mkdirs(f);
this.setPermission(f, permission);
return result;
}
@Override
public void setPermission(Path p, FsPermission permission)
throws IOException {
try {
super.setPermission(p, permission);
} catch (final IOException e) {
System.err.println("Cant help it, hence ignoring
IOException setting persmission for path \"" + p +
"\": " + e.getMessage());
}
}
}
This class was used in MapReduce Job as
if (RUN_LOCAL) {
conf.set("fs.default.name", "file:///");
conf.set("mapred.job.tracker", "local");
conf.set("fs.file.impl",
"org.scif.bdp.mrjobs.WindowsLocalFileSystem");
conf.set(
"io.serializations",
"org.apache.hadoop.io.serializer.JavaSerialization,"
+
"org.apache.hadoop.io.serializer.WritableSerialization");
}
It worked fine on CDH4.7. Now the same code when compiled on CDH5.1 works but
when I try to execute it throws the following stacktrace
Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:451)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:745)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:728)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633)
at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:467)
at
com.scif.bdp.common.WindowsLocalFileSystem.setPermission(WindowsLocalFileSystem.java:26)
at
com.scif.bdp.common.WindowsLocalFileSystem.mkdirs(WindowsLocalFileSystem.java:17)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
at com.scif.bdp.mrjobs.DeDup.run(DeDup.java:55)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.scif.bdp.mrjobs.DeDup.main(DeDup.java:59)
(Note DeDup is my MR class to remove duplicates)
Upon investigation the only change I saw was the change in method
.setPermission(). It invokes Native.POSIX.chmod as against Native.chmod
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[
https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096000#comment-14096000
]
Aaron T. Myers commented on HADOOP-10836:
-
Latest patch looks pretty good to me. I agree that no new tests are needed,
since this is really just a refactor. The fact that no test changes were
required is a nice confirmation that the refactor doesn't affect any actual
behavior.
Just one small question, +1 once this is addressed. Did you mean to leave this
property commented out of the httpfs-default.xml? Does the meaning or behavior
of this property change after this patch?
{code}
-
+
{code}
> Replace HttpFS custom proxyuser handling with common implementation
> ---
>
> Key: HADOOP-10836
> URL: https://issues.apache.org/jira/browse/HADOOP-10836
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch,
> HADOOP-10836.patch, HADOOP-10836.patch
>
>
> Use HADOOP-10835 to implement proxyuser logic in HttpFS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10904) Provide Alt to Clear Text Passwords through Cred Provider API
[ https://issues.apache.org/jira/browse/HADOOP-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095992#comment-14095992 ] Larry McCay commented on HADOOP-10904: -- Seems reasonable to me. I can't get to this till next week - if that's okay then we can assign a new jira to me. > Provide Alt to Clear Text Passwords through Cred Provider API > - > > Key: HADOOP-10904 > URL: https://issues.apache.org/jira/browse/HADOOP-10904 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > > This is an umbrella jira to track various child tasks to uptake the > credential provider API to enable deployments without storing > passwords/credentials in clear text. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095984#comment-14095984 ] Allen Wittenauer commented on HADOOP-8944: -- (Forgot to add the new Tests during the svn commit. Woops.. Build failures appear to be unrelated. Looks like a dependency problem in kms.) > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095978#comment-14095978 ] Hudson commented on HADOOP-8944: FAILURE: Integrated in Hadoop-trunk-Commit #6058 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6058/]) HADOOP-8944. Shell command fs -count should include human readable option (Jonathan Allen via aw) (continued) (aw: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617802) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestContentSummary.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095962#comment-14095962
]
Hadoop QA commented on HADOOP-10966:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661477/HADOOP-10966.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4467//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4467//console
This message is automatically generated.
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095943#comment-14095943 ] Alejandro Abdelnur commented on HADOOP-9902: Arpit, given the release notes, a bunch of incompatible changes. I've missed that before. So it cannot go in branch-2 as is, only trunk. My concern is that it will sit idle in trunk until a Hadoop 3 release. If others don't care about, well, I'm +0 on this. Ellen, you should have an explicit +1 before committing. Roman seems to have reviewed things in detail, I would ping him to stamp the +1. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9911) hadoop 2.1.0-beta tarball only contains 32bit native libraries
[ https://issues.apache.org/jira/browse/HADOOP-9911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095927#comment-14095927 ] Sean Jensen-Grey commented on HADOOP-9911: -- Why not ship the 64 bit library and leave the warning for the 32 bit users or ship both. But as it currently stands, shipping the 32 bit lib "breaks" a majority of the users. > hadoop 2.1.0-beta tarball only contains 32bit native libraries > -- > > Key: HADOOP-9911 > URL: https://issues.apache.org/jira/browse/HADOOP-9911 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.1.0-beta, 2.2.0 >Reporter: André Kelpe > > I am setting up a cluster on 64bit linux and I noticed, that the tarball only > ships with 32 bit libraries: > $ pwd > /opt/hadoop-2.1.0-beta/lib/native > $ ls -al > total 2376 > drwxr-xr-x 2 67974 users 4096 Aug 15 20:59 . > drwxr-xr-x 3 67974 users 4096 Aug 15 20:59 .. > -rw-r--r-- 1 67974 users 598578 Aug 15 20:59 libhadoop.a > -rw-r--r-- 1 67974 users 764772 Aug 15 20:59 libhadooppipes.a > lrwxrwxrwx 1 67974 users 18 Aug 15 20:59 libhadoop.so -> > libhadoop.so.1.0.0 > -rwxr-xr-x 1 67974 users 407568 Aug 15 20:59 libhadoop.so.1.0.0 > -rw-r--r-- 1 67974 users 304632 Aug 15 20:59 libhadooputils.a > -rw-r--r-- 1 67974 users 184414 Aug 15 20:59 libhdfs.a > lrwxrwxrwx 1 67974 users 16 Aug 15 20:59 libhdfs.so -> libhdfs.so.0.0.0 > -rwxr-xr-x 1 67974 users 149556 Aug 15 20:59 libhdfs.so.0.0.0 > $ file * > libhadoop.a:current ar archive > libhadooppipes.a: current ar archive > libhadoop.so: symbolic link to `libhadoop.so.1.0.0' > libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 > (SYSV), dynamically linked, > BuildID[sha1]=0x527e88ec3e92a95389839bd3fc9d7dbdebf654d6, not stripped > libhadooputils.a: current ar archive > libhdfs.a: current ar archive > libhdfs.so: symbolic link to `libhdfs.so.0.0.0' > libhdfs.so.0.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 > (SYSV), dynamically linked, > BuildID[sha1]=0xddb2abae9272f584edbe22c76b43fcda9436f685, not stripped > $ hadoop checknative > 13/08/28 18:11:17 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Native library checking: > hadoop: false > zlib: false > snappy: false > lz4:false > bzip2: false > 13/08/28 18:11:17 INFO util.ExitUtil: Exiting with status 1 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-10946: Attachment: HADOOP10946-03.patch Found one more typo. > Fix a bunch of typos in log messages > > > Key: HADOOP-10946 > URL: https://issues.apache.org/jira/browse/HADOOP-10946 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Ray Chiang >Priority: Trivial > Labels: newbie > Attachments: HADOOP10946-01.patch, HADOOP10946-02.patch, > HADOOP10946-03.patch > > > There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095913#comment-14095913 ] Arpit Agarwal commented on HADOOP-9902: --- bq. What are the key issues to be addressed for getting this in branch-2 and how can we take care of it? It would be good to know this before we commit to trunk. Thanks, Arpit. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-10946: Status: Open (was: Patch Available) Found one more spelling error. Will submit new patch. > Fix a bunch of typos in log messages > > > Key: HADOOP-10946 > URL: https://issues.apache.org/jira/browse/HADOOP-10946 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Ray Chiang >Priority: Trivial > Labels: newbie > Attachments: HADOOP10946-01.patch, HADOOP10946-02.patch, > HADOOP10946-03.patch > > > There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-10946: Status: Patch Available (was: Open) Submit for testing. > Fix a bunch of typos in log messages > > > Key: HADOOP-10946 > URL: https://issues.apache.org/jira/browse/HADOOP-10946 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Ray Chiang >Priority: Trivial > Labels: newbie > Attachments: HADOOP10946-01.patch, HADOOP10946-02.patch, > HADOOP10946-03.patch > > > There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10904) Provide Alt to Clear Text Passwords through Cred Provider API
[ https://issues.apache.org/jira/browse/HADOOP-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095901#comment-14095901 ] Andrew Wang commented on HADOOP-10904: -- We should fix this for KeyProvider too, since it currently does the same thing with repurposing the authority. Googling for "subscheme" didn't turn up much, but there is some precedent for using "+" like this, e.g. "svn+ssh". https://www.csoft.net/docs/svn.html.en > Provide Alt to Clear Text Passwords through Cred Provider API > - > > Key: HADOOP-10904 > URL: https://issues.apache.org/jira/browse/HADOOP-10904 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > > This is an umbrella jira to track various child tasks to uptake the > credential provider API to enable deployments without storing > passwords/credentials in clear text. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095891#comment-14095891 ] Allen Wittenauer commented on HADOOP-8989: -- Two things to fix: # The hdfs -find doc bug # Documentation should mention that the path parameter is optional and defaults to the user's home directory +1 lgtm, pending fixes to those two items. > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10962) Flags for posix_fadvise are not valid in some architectures
[ https://issues.apache.org/jira/browse/HADOOP-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095867#comment-14095867 ] Arpit Agarwal commented on HADOOP-10962: Unfortunately this broke Windows compilation but many thanks for the prompt fix [~dvillegas]! > Flags for posix_fadvise are not valid in some architectures > --- > > Key: HADOOP-10962 > URL: https://issues.apache.org/jira/browse/HADOOP-10962 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.4.1 > Environment: System z/Linux >Reporter: David Villegas >Assignee: David Villegas > Fix For: 2.6.0 > > Attachments: HADOOP-10962.patch > > > In org.apache.hadoop.io.nativeio.NativeIO.java, the posix_fadvise flag > parameter is hardcoded to the most common values in fcntl.h. > However, not all architectures use the same values (in this case, System > z/Linux) A better approach would be to not make assumptions about fcntl.h > values (or any other system constants). > This bug results in calls to posix_fadvise failing in zLinux. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-10966:
---
Status: Patch Available (was: Open)
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095866#comment-14095866
]
Arpit Agarwal commented on HADOOP-10966:
+1 pending Jenkins. Also verified this fixes the Windows build.
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10957) The globber will sometimes erroneously return a permission denied exception when there is a non-terminal wildcard
[ https://issues.apache.org/jira/browse/HADOOP-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095864#comment-14095864 ] Colin Patrick McCabe commented on HADOOP-10957: --- Let's just discuss the bug here, and discuss optimizations on HADOOP-10942. This is a tiny patch which fixes a significant bug, so hopefully we can get it in quickly. > The globber will sometimes erroneously return a permission denied exception > when there is a non-terminal wildcard > - > > Key: HADOOP-10957 > URL: https://issues.apache.org/jira/browse/HADOOP-10957 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10957.001.patch, HADOOP-10957.002.patch > > > The globber will sometimes erroneously return a permission denied exception > when there is a non-terminal wildcard. The existing unit tests don't catch > this, because it doesn't happen for superusers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10904) Provide Alt to Clear Text Passwords through Cred Provider API
[
https://issues.apache.org/jira/browse/HADOOP-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095855#comment-14095855
]
Daryn Sharp commented on HADOOP-10904:
--
Please fix {{CredentialProvider#unnestUri}}, in this jira or subtask, to not
violate URI syntax. Specifically:
bq. For example, "myscheme://hdfs@nn/my/path" is converted to
"hdfs://nn/my/path".
The "@" is RFC defined to separate userinfo and host:port in an authority.
I've been meaning to take advantage of the userinfo for a legit purpose (well
beyond the scope of this discussion) and an "abuse" like this will likely
conflict. A more URI-friendly approach is using subschemes:
"myscheme+hdfs://nn/my/path".
> Provide Alt to Clear Text Passwords through Cred Provider API
> -
>
> Key: HADOOP-10904
> URL: https://issues.apache.org/jira/browse/HADOOP-10904
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Larry McCay
>Assignee: Larry McCay
>
> This is an umbrella jira to track various child tasks to uptake the
> credential provider API to enable deployments without storing
> passwords/credentials in clear text.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10942) Globbing optimizations and regression fix
[
https://issues.apache.org/jira/browse/HADOOP-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095847#comment-14095847
]
Colin Patrick McCabe commented on HADOOP-10942:
---
bq. For the immediate file status, the prior code used to loop over the path
components even if there are globs. In this patch, it does an immediate file
status on the full path. This reduces the overhead for FsShell commands.
You always need to loop when there are globs. You need to see which children
match the glob and which don't. I think what you meant to write is "the prior
code used to loop over the path components even if there are *not* globs".
Looping is not a problem, though. Calling {{listStatus}} or {{fileStatus}} is
what generates RPCs. And the existing globber code doesn't do that unless it
needs to.
A simple way of seeing this is to add a LOG.info statement to
{{Globber#listStatus}} and {{Globber#getFileStatus}}, and then try {{hadoop fs
\-ls}} on a path without globs. The only output you will see is a single call
to {{getFileStatus}}, because that's the only call that's needed. The internal
looping that it does inside the function is not important because most loop
iterations don't generate an RPC.
> Globbing optimizations and regression fix
> -
>
> Key: HADOOP-10942
> URL: https://issues.apache.org/jira/browse/HADOOP-10942
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10942.patch
>
>
> When globbing was commonized to support both filesystem and filecontext, it
> regressed a fix that prevents an intermediate glob that matches a file from
> throwing a confusing permissions exception. The hdfs traverse check requires
> the exec bit which a file does not have.
> Additional optimizations to reduce rpcs actually increases them if
> directories contain 1 item.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095838#comment-14095838 ] Hudson commented on HADOOP-8944: FAILURE: Integrated in Hadoop-trunk-Commit #6056 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6056/]) HADOOP-8944. Shell command fs -count should include human readable option (Jonathan Allen via aw) (aw: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617775) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10843) TestGridmixRecord unit tests failure on PowerPC
[
https://issues.apache.org/jira/browse/HADOOP-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095833#comment-14095833
]
Hadoop QA commented on HADOOP-10843:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661350/HADOOP-10843.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-gridmix.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4466//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4466//console
This message is automatically generated.
> TestGridmixRecord unit tests failure on PowerPC
> ---
>
> Key: HADOOP-10843
> URL: https://issues.apache.org/jira/browse/HADOOP-10843
> Project: Hadoop Common
> Issue Type: Bug
> Components: test, tools
>Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.4.1
>Reporter: Jinghui Wang
>Assignee: Jinghui Wang
> Attachments: HADOOP-10843.1.patch, HADOOP-10843.2.patch,
> HADOOP-10843.3.patch
>
>
> In TestGridmixRecord#binSortTest, the test expects the comparison result of
> WritableComparator.compareBytes, which uses UnsafeComparer, to be the integer
> difference rather than the documented "@return 0 if equal, < 0 if left is
> less than right, etc.".
> TestGridmixRecord#binSortTest code snippet
> {code}
> final int chk = WritableComparator.compareBytes(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength());
> assertEquals(chk, x.compareTo(y));
> assertEquals(chk, cmp.compare(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength()));
> {code}
> The code snippet below shows the Unsafe comparator behavior for
> non-little-endian machines.
> {code}
> if (!littleEndian) {
> return lessThanUnsigned(lw, rw) ? -1 : 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095834#comment-14095834 ] Andrew Wang commented on HADOOP-9902: - Thanks for the work here Allen, +0 from me. > Shell script rewrite > > > Key: HADOOP-9902 > URL: https://issues.apache.org/jira/browse/HADOOP-9902 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: releasenotes > Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, > HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, > HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, > HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, > HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, > HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt > > > Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095822#comment-14095822
]
Allen Wittenauer commented on HADOOP-8989:
--
I've started to play with this patch. At first glance, I do see a (minor)
documentation bug:
{code}
+ Usage: << ... ... >>>
{code}
This should probably be:
{code}
+ Usage: << ... ... >>>
{code}
> hadoop dfs -find feature
>
>
> Key: HADOOP-8989
> URL: https://issues.apache.org/jira/browse/HADOOP-8989
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Marco Nicosia
>Assignee: Jonathan Allen
> Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch
>
>
> Both sysadmins and users make frequent use of the unix 'find' command, but
> Hadoop has no correlate. Without this, users are writing scripts which make
> heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
> -lsr is somewhat taxing on the NameNode, and a really slow experience on the
> client side. Possibly an in-NameNode find operation would be only a bit more
> taxing on the NameNode, but significantly faster from the client's point of
> view?
> The minimum set of options I can think of which would make a Hadoop find
> command generally useful is (in priority order):
> * -type (file or directory, for now)
> * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
> * -print0 (for piping to xargs -0)
> * -depth
> * -owner/-group (and -nouser/-nogroup)
> * -name (allowing for shell pattern, or even regex?)
> * -perm
> * -size
> One possible special case, but could possibly be really cool if it ran from
> within the NameNode:
> * -delete
> The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow.
> Lower priority, some people do use operators, mostly to execute -or searches
> such as:
> * find / \(-nouser -or -nogroup\)
> Finally, I thought I'd include a link to the [Posix spec for
> find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095806#comment-14095806
]
Vinayakumar B commented on HADOOP-10966:
Thanks [~dvillegas],
Current patch fixes the compilation.
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HADOOP-10966:
---
Assignee: David Villegas
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Assignee: David Villegas
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-7713) dfs -count -q should label output column
[
https://issues.apache.org/jira/browse/HADOOP-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095803#comment-14095803
]
Daryn Sharp commented on HADOOP-7713:
-
Like the other count jira, this would be great to have in 2.x. I know we have
various SE and customer monitoring scripts that depend on this command, so I'd
rather not have the headers by default so we don't have to hunt down everyone
using the command.
I think the header field names in count's former help message are probably a
better choice than the new ones in this patch. I don't mind the upper/lower
case so much as fields that contain spaces. I often like to write utilities
that split a header and use them as map keys. Spaces in the header fields
thwarts that.
I'd suggest using a single letter for the header option (maybe v(erbose)) to be
inline with the other commands. Otherwise it will thwart the "one of these
days" feature of allowing option bundling. Multi-char options are better
implemented as "--opt".
> dfs -count -q should label output column
>
>
> Key: HADOOP-7713
> URL: https://issues.apache.org/jira/browse/HADOOP-7713
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nigel Daley
>Assignee: Jonathan Allen
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
> HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch
>
>
> These commands should label the output columns:
> {code}
> hadoop dfs -count ...
> hadoop dfs -count -q ...
> {code}
> Current output of the 2nd command above:
> {code}
> % hadoop dfs -count -q /user/foo /tmp
> none inf 9569 9493 6372553322
> hdfs://nn1.bar.com/user/foo
> none inf 101 2689 209349812906
> hdfs://nn1.bar.com/tmp
> {code}
> It is not obvious what these columns mean.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Issue Comment Deleted] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-8944: --- Comment: was deleted (was: Like the other count jira, this would be great to have in 2.x. I know we have various SE and customer monitoring scripts that depend on this command, so I'd rather not have the headers by default so we don't have to hunt down everyone using the command. I think the header field names in count's former help message are probably a better choice than the new ones in this patch. I don't mind the upper/lower case so much as fields that contain spaces. I often like to write utilities that split a header and use them as map keys. Spaces in the header fields thwarts that. I'd suggest using a single letter for the header option (maybe v(erbose)) to be inline with the other commands. Otherwise it will thwart the "one of these days" feature of allowing option bundling. Multi-char options are better implemented as "--opt".) > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Issue Comment Deleted] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-8944: --- Comment: was deleted (was: wrong jira, ignore) > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095802#comment-14095802 ] Daryn Sharp commented on HADOOP-8944: - wrong jira, ignore > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095801#comment-14095801 ] Daryn Sharp commented on HADOOP-8944: - Like the other count jira, this would be great to have in 2.x. I know we have various SE and customer monitoring scripts that depend on this command, so I'd rather not have the headers by default so we don't have to hunt down everyone using the command. I think the header field names in count's former help message are probably a better choice than the new ones in this patch. I don't mind the upper/lower case so much as fields that contain spaces. I often like to write utilities that split a header and use them as map keys. Spaces in the header fields thwarts that. I'd suggest using a single letter for the header option (maybe v(erbose)) to be inline with the other commands. Otherwise it will thwart the "one of these days" feature of allowing option bundling. Multi-char options are better implemented as "--opt". > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-8944: - Fix Version/s: 2.6.0 3.0.0 > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8944) Shell command fs -count should include human readable option
[ https://issues.apache.org/jira/browse/HADOOP-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-8944: - Resolution: Fixed Status: Resolved (was: Patch Available) This has been committed to branch-2 and trunk. Huzzah! > Shell command fs -count should include human readable option > > > Key: HADOOP-8944 > URL: https://issues.apache.org/jira/browse/HADOOP-8944 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jonathan Allen >Assignee: Allen Wittenauer >Priority: Trivial > Labels: incompatible, newbie > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-8944-1.patch, HADOOP-8944-2.patch, > HADOOP-8944-3.patch, HADOOP-8944-4.patch, HADOOP-8944.patch > > > The shell command fs -count report sizes in bytes. The command should accept > a -h option to display the sizes in a human readable format, i.e. K, M, G, > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[
https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095747#comment-14095747
]
Hadoop QA commented on HADOOP-10968:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661482/0001-Set-java_libarch-for-ppc64le-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.ha.TestZKFailoverControllerStress
org.apache.hadoop.ipc.TestDecayRpcScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4465//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4465//console
This message is automatically generated.
> hadoop common fails to detect java_libarch on ppc64le
> -
>
> Key: HADOOP-10968
> URL: https://issues.apache.org/jira/browse/HADOOP-10968
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.3.0
>Reporter: Dinar Valeev
> Fix For: 0.23.2
>
> Attachments: 0001-Set-java_libarch-for-ppc64le-2.patch
>
>
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common ---
> [INFO] Executing tasks
> main:
> [exec] -- The C compiler identification is GNU 4.8.3
> [exec] -- The CXX compiler identification is GNU 4.8.3
> [exec] -- Check for working C compiler: /usr/bin/cc
> [exec] -- Check for working C compiler: /usr/bin/cc -- works
> [exec] -- Detecting C compiler ABI info
> [exec] -- Detecting C compiler ABI info - done
> [exec] -- Check for working CXX compiler: /usr/bin/c++
> [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works
> [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND
> [exec]
> JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include,
> JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux
> [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE):
> [exec] Failed to find a viable JVM installation under JAVA_HOME.
> [exec] Call Stack (most recent call first):
> [exec] CMakeLists.txt:24 (include)
> [exec]
> [exec]
> [exec] -- Detecting CXX compiler ABI info
> [exec] -- Detecting CXX compiler ABI info - done
> [exec] -- Configuring incomplete, errors occurred!
> [exec] See also
> "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log".
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main . SUCCESS [ 10.680
> s]
> [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716
> s]
> [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270
> s]
> [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274
> s]
> [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819
> s]
> [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284
> s]
> [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863
> s]
> [INFO] Apache Hadoop Auth . SUCCESS [ 4.032
> s]
> [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475
> s]
> [INFO] Apache Hadoop Common ... FAILURE [ 10.458
> s]
> [INFO] Apa
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095744#comment-14095744 ] Sanjay Radia commented on HADOOP-10919: --- bq. trashing It's assumed that an hdfs admin would not (intentionally) do that. Okay, please add that your doc when you next update it. We could allow just read access to /r/r/ to all. Use cases: charles can we please work together to get the distcp use cases nailed. We can work offline to go faster and then summarize for the community. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10843) TestGridmixRecord unit tests failure on PowerPC
[
https://issues.apache.org/jira/browse/HADOOP-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinghui Wang updated HADOOP-10843:
--
Target Version/s: (was: 2.5.0)
Status: Patch Available (was: Open)
> TestGridmixRecord unit tests failure on PowerPC
> ---
>
> Key: HADOOP-10843
> URL: https://issues.apache.org/jira/browse/HADOOP-10843
> Project: Hadoop Common
> Issue Type: Bug
> Components: test, tools
>Affects Versions: 2.4.1, 2.4.0, 2.3.0, 2.2.0
>Reporter: Jinghui Wang
>Assignee: Jinghui Wang
> Attachments: HADOOP-10843.1.patch, HADOOP-10843.2.patch,
> HADOOP-10843.3.patch
>
>
> In TestGridmixRecord#binSortTest, the test expects the comparison result of
> WritableComparator.compareBytes, which uses UnsafeComparer, to be the integer
> difference rather than the documented "@return 0 if equal, < 0 if left is
> less than right, etc.".
> TestGridmixRecord#binSortTest code snippet
> {code}
> final int chk = WritableComparator.compareBytes(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength());
> assertEquals(chk, x.compareTo(y));
> assertEquals(chk, cmp.compare(
> out1.getData(), 0, out1.getLength(),
> out2.getData(), 0, out2.getLength()));
> {code}
> The code snippet below shows the Unsafe comparator behavior for
> non-little-endian machines.
> {code}
> if (!littleEndian) {
> return lessThanUnsigned(lw, rw) ? -1 : 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10959) A Complement and Short Term Solution to TokenAuth Based on Kerberos Pre-Authentication Framework
[ https://issues.apache.org/jira/browse/HADOOP-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095693#comment-14095693 ] Daryn Sharp commented on HADOOP-10959: -- I digress, but I'm not sure why generating principals and keytabs seems to be viewed as a difficult activity. Most if not nearly all large corporations already use kerberos in some form. Here's a few initial thoughts/questions: My impression it's a misnomer to call this a pre-auth extension? It's not a hardening like the encrypted timestamp pre-auth, but instead is a complete substitute for a password/keytab. Doesn't this completely undermine kerberos? Will cross-realm authentication work? Do all KDCs in the trust have to use the custom plugin? I probably overlooked it, but how do tasks running in the cluster authenticate under the model? Do they continue to use the existing delegation tokens obtained via JWT/TGT during job submission, or are they using the JWT tokens to obtain a TGT/TGS in the tasks? I think the latter? The doc says having the NN configured for user to group mappings is not ideal. Mention is made of an AD-TOKEN which I believe is a non-standard MS extension? Do you envision the JWT issuer containing the group mapping for all services? bq. However, the pain to deploy keytabs for services can be alleviated by token support, still, another story. I'm not sure how you remove the "pain" of copying a file. For mutual auth, I would expect something like certificates or private keys or shared secret to be involved - which would be an equal but different "pain to deploy"? > A Complement and Short Term Solution to TokenAuth Based on Kerberos > Pre-Authentication Framework > > > Key: HADOOP-10959 > URL: https://issues.apache.org/jira/browse/HADOOP-10959 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: Rhino > Attachments: KerbToken-v2.pdf > > > To implement and integrate pluggable authentication providers, enhance > desirable single sign on for end users, and help enforce centralized access > control on the platform, the community has widely discussed and concluded > token based authentication could be the appropriate approach. TokenAuth > (HADOOP-9392) was proposed and is under development to implement another > Authentication Method in lieu with Simple and Kerberos. It is a big and long > term effort to support TokenAuth across the entire ecosystem. We here propose > a short term replacement based on Kerberos that can complement to TokenAuth. > Our solution involves less codes changes with limited risk and the main > development work has already been done in our POC. Users can use our solution > as a short term solution to support token inside Hadoop. > This effort and resultant solution will be fully described in the design > document to be attached. And the brief introduction will be commented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[ https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinar Valeev updated HADOOP-10968: -- Attachment: (was: 0001-Set-java_libarch-for-ppc64le-1.patch) > hadoop common fails to detect java_libarch on ppc64le > - > > Key: HADOOP-10968 > URL: https://issues.apache.org/jira/browse/HADOOP-10968 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.3.0 >Reporter: Dinar Valeev > Fix For: 0.23.2 > > Attachments: 0001-Set-java_libarch-for-ppc64le-2.patch > > > [INFO] > [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common --- > [INFO] Executing tasks > main: > [exec] -- The C compiler identification is GNU 4.8.3 > [exec] -- The CXX compiler identification is GNU 4.8.3 > [exec] -- Check for working C compiler: /usr/bin/cc > [exec] -- Check for working C compiler: /usr/bin/cc -- works > [exec] -- Detecting C compiler ABI info > [exec] -- Detecting C compiler ABI info - done > [exec] -- Check for working CXX compiler: /usr/bin/c++ > [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works > [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND > [exec] > JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include, > JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux > [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE): > [exec] Failed to find a viable JVM installation under JAVA_HOME. > [exec] Call Stack (most recent call first): > [exec] CMakeLists.txt:24 (include) > [exec] > [exec] > [exec] -- Detecting CXX compiler ABI info > [exec] -- Detecting CXX compiler ABI info - done > [exec] -- Configuring incomplete, errors occurred! > [exec] See also > "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log". > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ 10.680 > s] > [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716 > s] > [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270 > s] > [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274 > s] > [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819 > s] > [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284 > s] > [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863 > s] > [INFO] Apache Hadoop Auth . SUCCESS [ 4.032 > s] > [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475 > s] > [INFO] Apache Hadoop Common ... FAILURE [ 10.458 > s] > [INFO] Apache Hadoop NFS .. SKIPPED > [INFO] Apache Hadoop Common Project ... SKIPPED > [INFO] Apache Hadoop HDFS . SKIPPED > [INFO] Apache Hadoop HttpFS ... SKIPPED > [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED > [INFO] Apache Hadoop HDFS-NFS . SKIPPED > [INFO] Apache Hadoop HDFS Project . SKIPPED > [INFO] hadoop-yarn SKIPPED > [INFO] hadoop-yarn-api SKIPPED > [INFO] hadoop-yarn-common . SKIPPED > [INFO] hadoop-yarn-server . SKIPPED > [INFO] hadoop-yarn-server-common .. SKIPPED > [INFO] hadoop-yarn-server-nodemanager . SKIPPED > [INFO] hadoop-yarn-server-web-proxy ... SKIPPED > [INFO] hadoop-yarn-server-resourcemanager . SKIPPED > [INFO] hadoop-yarn-server-tests ... SKIPPED > [INFO] hadoop-yarn-client . SKIPPED > [INFO] hadoop-yarn-applications ... SKIPPED > [INFO] hadoop-yarn-applications-distributedshell .. SKIPPED > [INFO] hadoop-yarn-applications-unmanaged-am-launcher . SKIPPED > [INFO] hadoop-yarn-site ... SKIPPED > [INFO] hadoop-yarn-project SKIPPED > [INFO] hadoop-mapreduce-client SKIPPED > [INFO] hadoop-mapreduce-client-core ... SKIPPED > [INFO] hadoop-mapreduce-client-common . SKIPPED > [INFO] hadoop-mapreduce-client-shuffle SKIPPED > [INFO] hadoop-mapreduce-client-app ..
[jira] [Updated] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[ https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinar Valeev updated HADOOP-10968: -- Attachment: 0001-Set-java_libarch-for-ppc64le-2.patch Fixed cmake syntax > hadoop common fails to detect java_libarch on ppc64le > - > > Key: HADOOP-10968 > URL: https://issues.apache.org/jira/browse/HADOOP-10968 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.3.0 >Reporter: Dinar Valeev > Fix For: 0.23.2 > > Attachments: 0001-Set-java_libarch-for-ppc64le-2.patch > > > [INFO] > [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common --- > [INFO] Executing tasks > main: > [exec] -- The C compiler identification is GNU 4.8.3 > [exec] -- The CXX compiler identification is GNU 4.8.3 > [exec] -- Check for working C compiler: /usr/bin/cc > [exec] -- Check for working C compiler: /usr/bin/cc -- works > [exec] -- Detecting C compiler ABI info > [exec] -- Detecting C compiler ABI info - done > [exec] -- Check for working CXX compiler: /usr/bin/c++ > [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works > [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND > [exec] > JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include, > JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux > [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE): > [exec] Failed to find a viable JVM installation under JAVA_HOME. > [exec] Call Stack (most recent call first): > [exec] CMakeLists.txt:24 (include) > [exec] > [exec] > [exec] -- Detecting CXX compiler ABI info > [exec] -- Detecting CXX compiler ABI info - done > [exec] -- Configuring incomplete, errors occurred! > [exec] See also > "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log". > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ 10.680 > s] > [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716 > s] > [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270 > s] > [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274 > s] > [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819 > s] > [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284 > s] > [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863 > s] > [INFO] Apache Hadoop Auth . SUCCESS [ 4.032 > s] > [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475 > s] > [INFO] Apache Hadoop Common ... FAILURE [ 10.458 > s] > [INFO] Apache Hadoop NFS .. SKIPPED > [INFO] Apache Hadoop Common Project ... SKIPPED > [INFO] Apache Hadoop HDFS . SKIPPED > [INFO] Apache Hadoop HttpFS ... SKIPPED > [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED > [INFO] Apache Hadoop HDFS-NFS . SKIPPED > [INFO] Apache Hadoop HDFS Project . SKIPPED > [INFO] hadoop-yarn SKIPPED > [INFO] hadoop-yarn-api SKIPPED > [INFO] hadoop-yarn-common . SKIPPED > [INFO] hadoop-yarn-server . SKIPPED > [INFO] hadoop-yarn-server-common .. SKIPPED > [INFO] hadoop-yarn-server-nodemanager . SKIPPED > [INFO] hadoop-yarn-server-web-proxy ... SKIPPED > [INFO] hadoop-yarn-server-resourcemanager . SKIPPED > [INFO] hadoop-yarn-server-tests ... SKIPPED > [INFO] hadoop-yarn-client . SKIPPED > [INFO] hadoop-yarn-applications ... SKIPPED > [INFO] hadoop-yarn-applications-distributedshell .. SKIPPED > [INFO] hadoop-yarn-applications-unmanaged-am-launcher . SKIPPED > [INFO] hadoop-yarn-site ... SKIPPED > [INFO] hadoop-yarn-project SKIPPED > [INFO] hadoop-mapreduce-client SKIPPED > [INFO] hadoop-mapreduce-client-core ... SKIPPED > [INFO] hadoop-mapreduce-client-common . SKIPPED > [INFO] hadoop-mapreduce-client-shuffle SKIPPED > [INFO] hadoop-mapreduce-client-app .
[jira] [Commented] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[
https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095631#comment-14095631
]
Hadoop QA commented on HADOOP-10968:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661469/0001-Set-java_libarch-for-ppc64le-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4463//console
This message is automatically generated.
> hadoop common fails to detect java_libarch on ppc64le
> -
>
> Key: HADOOP-10968
> URL: https://issues.apache.org/jira/browse/HADOOP-10968
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.3.0
>Reporter: Dinar Valeev
> Fix For: 0.23.2
>
> Attachments: 0001-Set-java_libarch-for-ppc64le-1.patch
>
>
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common ---
> [INFO] Executing tasks
> main:
> [exec] -- The C compiler identification is GNU 4.8.3
> [exec] -- The CXX compiler identification is GNU 4.8.3
> [exec] -- Check for working C compiler: /usr/bin/cc
> [exec] -- Check for working C compiler: /usr/bin/cc -- works
> [exec] -- Detecting C compiler ABI info
> [exec] -- Detecting C compiler ABI info - done
> [exec] -- Check for working CXX compiler: /usr/bin/c++
> [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works
> [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND
> [exec]
> JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include,
> JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux
> [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE):
> [exec] Failed to find a viable JVM installation under JAVA_HOME.
> [exec] Call Stack (most recent call first):
> [exec] CMakeLists.txt:24 (include)
> [exec]
> [exec]
> [exec] -- Detecting CXX compiler ABI info
> [exec] -- Detecting CXX compiler ABI info - done
> [exec] -- Configuring incomplete, errors occurred!
> [exec] See also
> "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log".
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main . SUCCESS [ 10.680
> s]
> [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716
> s]
> [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270
> s]
> [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274
> s]
> [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819
> s]
> [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284
> s]
> [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863
> s]
> [INFO] Apache Hadoop Auth . SUCCESS [ 4.032
> s]
> [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475
> s]
> [INFO] Apache Hadoop Common ... FAILURE [ 10.458
> s]
> [INFO] Apache Hadoop NFS .. SKIPPED
> [INFO] Apache Hadoop Common Project ... SKIPPED
> [INFO] Apache Hadoop HDFS . SKIPPED
> [INFO] Apache Hadoop HttpFS ... SKIPPED
> [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED
> [INFO] Apache Hadoop HDFS-NFS . SKIPPED
> [INFO] Apache Hadoop HDFS Project . SKIPPED
> [INFO] hadoop-yarn SKIPPED
> [INFO] hadoop-yarn-api SKIPPED
> [INFO] hadoop-yarn-common . SKIPPED
> [INFO] hadoop-yarn-server . SKIPPED
> [INFO] hadoop-yarn-server-common .. SKIPPED
> [INFO] hadoop-yarn-server-nodemanager . SKIPPED
> [INFO] hadoop-yarn-server-web-proxy ... SKIPPED
> [INFO] hadoop-yarn-server-resourcemanager . SKIPPED
> [INFO] hadoop-yarn-server-tests .
[jira] [Commented] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095632#comment-14095632
]
David Villegas commented on HADOOP-10966:
-
Attached a patch to address the issue. I don't have a Windows build
environment, so I'd appreciate any help validating it.
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10966) Hadoop Common native compilation broken in windows
[
https://issues.apache.org/jira/browse/HADOOP-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Villegas updated HADOOP-10966:
Attachment: HADOOP-10966.patch
> Hadoop Common native compilation broken in windows
> --
>
> Key: HADOOP-10966
> URL: https://issues.apache.org/jira/browse/HADOOP-10966
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Reporter: Vinayakumar B
>Priority: Blocker
> Attachments: HADOOP-10966.patch
>
>
> After HADOOP-10962 hadoop common native compilation broken in windows
> {noformat}
> src\org\apache\hadoop\io\nativeio\NativeIO.c(181): error C2065:
> 'POSIX_FADV_NORMAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(184): error C2065:
> 'POSIX_FADV_RANDOM' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(187): error C2065:
> 'POSIX_FADV_SEQUENTIAL' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(190): error C2065:
> 'POSIX_FADV_WILLNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(193): error C2065:
> 'POSIX_FADV_DONTNEED' : undeclared identifier
> src\org\apache\hadoop\io\nativeio\NativeIO.c(196): error C2065:
> 'POSIX_FADV_NOREUSE' : undeclared identifier
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-7713) dfs -count -q should label output column
[
https://issues.apache.org/jira/browse/HADOOP-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095622#comment-14095622
]
Hadoop QA commented on HADOOP-7713:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554752/HADOOP-7713.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4464//console
This message is automatically generated.
> dfs -count -q should label output column
>
>
> Key: HADOOP-7713
> URL: https://issues.apache.org/jira/browse/HADOOP-7713
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nigel Daley
>Assignee: Jonathan Allen
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
> HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch
>
>
> These commands should label the output columns:
> {code}
> hadoop dfs -count ...
> hadoop dfs -count -q ...
> {code}
> Current output of the 2nd command above:
> {code}
> % hadoop dfs -count -q /user/foo /tmp
> none inf 9569 9493 6372553322
> hdfs://nn1.bar.com/user/foo
> none inf 101 2689 209349812906
> hdfs://nn1.bar.com/tmp
> {code}
> It is not obvious what these columns mean.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10851) NetgroupCache does not remove group memberships
[ https://issues.apache.org/jira/browse/HADOOP-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095593#comment-14095593 ] Hudson commented on HADOOP-10851: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1862 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1862/]) HADOOP-10851. NetgroupCache does not remove group memberships. (Contributed by Benoy Antony) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617612) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/NetgroupCache.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNetgroupCache.java > NetgroupCache does not remove group memberships > --- > > Key: HADOOP-10851 > URL: https://issues.apache.org/jira/browse/HADOOP-10851 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-10851.patch > > > _NetgroupCache_ is used by _GroupMappingServiceProvider_ implementations > based on net groups. > But it has a serious flaw in that once a user to group membership is > established, it remains forever even if user is actually removed from the > netgroup and cache is cleared. It is cleared only if the server is restarted. > To reproduce this: > * Cache a group with a set of users. > * Test membership correctness. > * Clear cache, remove a user from the group and cache the group again > * Expected result : user’s groups should not include the group from which > he/she is removed. > * Actual result : user’s groups includes the group from which he/she was > removed. > It is also noted that _NetgroupCache_ has concurrency issues and a separate > jira is filed to rectify them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10281) Create a scheduler, which assigns schedulables a priority level
[ https://issues.apache.org/jira/browse/HADOOP-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095595#comment-14095595 ] Hudson commented on HADOOP-10281: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1862 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1862/]) HADOOP-10281. Create a scheduler, which assigns schedulables a priority level. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617643) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcSchedulerMXBean.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcScheduler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestDecayRpcScheduler.java > Create a scheduler, which assigns schedulables a priority level > --- > > Key: HADOOP-10281 > URL: https://issues.apache.org/jira/browse/HADOOP-10281 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li >Assignee: Chris Li > Fix For: 3.0.0, 2.6.0 > > Attachments: HADOOP-10281-preview.patch, HADOOP-10281.patch, > HADOOP-10281.patch, HADOOP-10281.patch, HADOOP-10281.patch, HADOOP-10281.patch > > > The Scheduler decides which sub-queue to assign a given Call. It implements a > single method getPriorityLevel(Schedulable call) which returns an integer > corresponding to the subqueue the FairCallQueue should place the call in. > The HistoryRpcScheduler is one such implementation which uses the username of > each call and determines what % of calls in recent history were made by this > user. > It is configured with a historyLength (how many calls to track) and a list of > integer thresholds which determine the boundaries between priority levels. > For instance, if the scheduler has a historyLength of 8; and priority > thresholds of 4,2,1; and saw calls made by these users in order: > Alice, Bob, Alice, Alice, Bob, Jerry, Alice, Alice > * Another call by Alice would be placed in queue 3, since she has already > made >= 4 calls > * Another call by Bob would be placed in queue 2, since he has >= 2 but less > than 4 calls > * A call by Carlos would be placed in queue 0, since he has no calls in the > history > Also, some versions of this patch include the concept of a 'service user', > which is a user that is always scheduled high-priority. Currently this seems > redundant and will probably be removed in later patches, since its not too > useful. > > As of now, the current scheduler is the DecayRpcScheduler, which only keeps > track of the number of each type of call and decays these counts periodically. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10962) Flags for posix_fadvise are not valid in some architectures
[ https://issues.apache.org/jira/browse/HADOOP-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095601#comment-14095601 ] Hudson commented on HADOOP-10962: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1862 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1862/]) HADOOP-10962. Flags for posix_fadvise are not valid in some architectures (David Villegas via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617621) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c > Flags for posix_fadvise are not valid in some architectures > --- > > Key: HADOOP-10962 > URL: https://issues.apache.org/jira/browse/HADOOP-10962 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.4.1 > Environment: System z/Linux >Reporter: David Villegas >Assignee: David Villegas > Fix For: 2.6.0 > > Attachments: HADOOP-10962.patch > > > In org.apache.hadoop.io.nativeio.NativeIO.java, the posix_fadvise flag > parameter is hardcoded to the most common values in fcntl.h. > However, not all architectures use the same values (in this case, System > z/Linux) A better approach would be to not make assumptions about fcntl.h > values (or any other system constants). > This bug results in calls to posix_fadvise failing in zLinux. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10964) Small fix for NetworkTopologyWithNodeGroup#sortByDistance
[
https://issues.apache.org/jira/browse/HADOOP-10964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095604#comment-14095604
]
Hadoop QA commented on HADOOP-10964:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661443/HADOOP-10964.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverControllerStress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4462//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4462//console
This message is automatically generated.
> Small fix for NetworkTopologyWithNodeGroup#sortByDistance
> -
>
> Key: HADOOP-10964
> URL: https://issues.apache.org/jira/browse/HADOOP-10964
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Minor
> Attachments: HADOOP-10964.001.patch
>
>
> {{nodes.length}} should be {{activeLen}}.
> {code}
> @Override
> public void sortByDistance(Node reader, Node[] nodes, int activeLen,
> long seed, boolean randomizeBlockLocationsPerBlock) {
> // If reader is not a datanode (not in NetworkTopology tree), we need to
> // replace this reader with a sibling leaf node in tree.
> if (reader != null && !this.contains(reader)) {
> Node nodeGroup = getNode(reader.getNetworkLocation());
> if (nodeGroup != null && nodeGroup instanceof InnerNode) {
> InnerNode parentNode = (InnerNode) nodeGroup;
> // replace reader with the first children of its parent in tree
> reader = parentNode.getLeaf(0, null);
> } else {
> return;
> }
> }
> super.sortByDistance(reader, nodes, nodes.length, seed,
> randomizeBlockLocationsPerBlock);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10820) Throw an exception in GenericOptionsParser when passed an empty Path
[
https://issues.apache.org/jira/browse/HADOOP-10820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095602#comment-14095602
]
Hudson commented on HADOOP-10820:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1862 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1862/])
HADOOP-10820. Throw an exception in GenericOptionsParser when passed an empty
Path. Contributed by Alex Holmes and Zhihai Xu. (wang:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1617542)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
> Throw an exception in GenericOptionsParser when passed an empty Path
>
>
> Key: HADOOP-10820
> URL: https://issues.apache.org/jira/browse/HADOOP-10820
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Alex Holmes
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.6.0
>
> Attachments: HADOOP-10820-1.patch, HADOOP-10820-2.patch,
> HADOOP-10820-3.patch, HADOOP-10820.patch
>
>
> An empty token (e.g. "a.jar,,b.jar") in the -libjars option causes the
> current working directory to be recursively localized.
> Here's an example of this in action (using Hadoop 2.2.0):
> {code}
> # create a temp directory and touch three JAR files
> mkdir -p tmp/path && cd tmp && touch a.jar b.jar c.jar path/d.jar
> # Run an example job only specifying two of the JARs.
> # Include an empty entry in libjars.
> hadoop jar
> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> pi -libjars a.jar,,c.jar 2 10
> # As the job is running examine the localized directory in HDFS.
> # Notice that not only are the two JAR's specified in libjars copied,
> # but in addition the contents of the working directory are also recursively
> copied.
> $ hadoop fs -lsr
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/a.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/c.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/a.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/b.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/c.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/path
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/path/d.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[ https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinar Valeev updated HADOOP-10968: -- Fix Version/s: 0.23.2 Target Version/s: 2.4.1 Status: Patch Available (was: Open) > hadoop common fails to detect java_libarch on ppc64le > - > > Key: HADOOP-10968 > URL: https://issues.apache.org/jira/browse/HADOOP-10968 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.3.0 >Reporter: Dinar Valeev > Fix For: 0.23.2 > > Attachments: 0001-Set-java_libarch-for-ppc64le-1.patch > > > [INFO] > [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common --- > [INFO] Executing tasks > main: > [exec] -- The C compiler identification is GNU 4.8.3 > [exec] -- The CXX compiler identification is GNU 4.8.3 > [exec] -- Check for working C compiler: /usr/bin/cc > [exec] -- Check for working C compiler: /usr/bin/cc -- works > [exec] -- Detecting C compiler ABI info > [exec] -- Detecting C compiler ABI info - done > [exec] -- Check for working CXX compiler: /usr/bin/c++ > [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works > [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND > [exec] > JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include, > JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux > [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE): > [exec] Failed to find a viable JVM installation under JAVA_HOME. > [exec] Call Stack (most recent call first): > [exec] CMakeLists.txt:24 (include) > [exec] > [exec] > [exec] -- Detecting CXX compiler ABI info > [exec] -- Detecting CXX compiler ABI info - done > [exec] -- Configuring incomplete, errors occurred! > [exec] See also > "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log". > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ 10.680 > s] > [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716 > s] > [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270 > s] > [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274 > s] > [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819 > s] > [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284 > s] > [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863 > s] > [INFO] Apache Hadoop Auth . SUCCESS [ 4.032 > s] > [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475 > s] > [INFO] Apache Hadoop Common ... FAILURE [ 10.458 > s] > [INFO] Apache Hadoop NFS .. SKIPPED > [INFO] Apache Hadoop Common Project ... SKIPPED > [INFO] Apache Hadoop HDFS . SKIPPED > [INFO] Apache Hadoop HttpFS ... SKIPPED > [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED > [INFO] Apache Hadoop HDFS-NFS . SKIPPED > [INFO] Apache Hadoop HDFS Project . SKIPPED > [INFO] hadoop-yarn SKIPPED > [INFO] hadoop-yarn-api SKIPPED > [INFO] hadoop-yarn-common . SKIPPED > [INFO] hadoop-yarn-server . SKIPPED > [INFO] hadoop-yarn-server-common .. SKIPPED > [INFO] hadoop-yarn-server-nodemanager . SKIPPED > [INFO] hadoop-yarn-server-web-proxy ... SKIPPED > [INFO] hadoop-yarn-server-resourcemanager . SKIPPED > [INFO] hadoop-yarn-server-tests ... SKIPPED > [INFO] hadoop-yarn-client . SKIPPED > [INFO] hadoop-yarn-applications ... SKIPPED > [INFO] hadoop-yarn-applications-distributedshell .. SKIPPED > [INFO] hadoop-yarn-applications-unmanaged-am-launcher . SKIPPED > [INFO] hadoop-yarn-site ... SKIPPED > [INFO] hadoop-yarn-project SKIPPED > [INFO] hadoop-mapreduce-client SKIPPED > [INFO] hadoop-mapreduce-client-core ... SKIPPED > [INFO] hadoop-mapreduce-client-common . SKIPPED > [INFO] hadoop-mapreduce-client-shuffle SKIPPED > [INFO] had
[jira] [Updated] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[ https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinar Valeev updated HADOOP-10968: -- Attachment: (was: 0001-Set-java_libarch-for-ppc64le.patch) > hadoop common fails to detect java_libarch on ppc64le > - > > Key: HADOOP-10968 > URL: https://issues.apache.org/jira/browse/HADOOP-10968 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.3.0 >Reporter: Dinar Valeev > Attachments: 0001-Set-java_libarch-for-ppc64le-1.patch > > > [INFO] > [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common --- > [INFO] Executing tasks > main: > [exec] -- The C compiler identification is GNU 4.8.3 > [exec] -- The CXX compiler identification is GNU 4.8.3 > [exec] -- Check for working C compiler: /usr/bin/cc > [exec] -- Check for working C compiler: /usr/bin/cc -- works > [exec] -- Detecting C compiler ABI info > [exec] -- Detecting C compiler ABI info - done > [exec] -- Check for working CXX compiler: /usr/bin/c++ > [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works > [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND > [exec] > JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include, > JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux > [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE): > [exec] Failed to find a viable JVM installation under JAVA_HOME. > [exec] Call Stack (most recent call first): > [exec] CMakeLists.txt:24 (include) > [exec] > [exec] > [exec] -- Detecting CXX compiler ABI info > [exec] -- Detecting CXX compiler ABI info - done > [exec] -- Configuring incomplete, errors occurred! > [exec] See also > "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log". > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ 10.680 > s] > [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716 > s] > [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270 > s] > [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274 > s] > [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819 > s] > [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284 > s] > [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863 > s] > [INFO] Apache Hadoop Auth . SUCCESS [ 4.032 > s] > [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475 > s] > [INFO] Apache Hadoop Common ... FAILURE [ 10.458 > s] > [INFO] Apache Hadoop NFS .. SKIPPED > [INFO] Apache Hadoop Common Project ... SKIPPED > [INFO] Apache Hadoop HDFS . SKIPPED > [INFO] Apache Hadoop HttpFS ... SKIPPED > [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED > [INFO] Apache Hadoop HDFS-NFS . SKIPPED > [INFO] Apache Hadoop HDFS Project . SKIPPED > [INFO] hadoop-yarn SKIPPED > [INFO] hadoop-yarn-api SKIPPED > [INFO] hadoop-yarn-common . SKIPPED > [INFO] hadoop-yarn-server . SKIPPED > [INFO] hadoop-yarn-server-common .. SKIPPED > [INFO] hadoop-yarn-server-nodemanager . SKIPPED > [INFO] hadoop-yarn-server-web-proxy ... SKIPPED > [INFO] hadoop-yarn-server-resourcemanager . SKIPPED > [INFO] hadoop-yarn-server-tests ... SKIPPED > [INFO] hadoop-yarn-client . SKIPPED > [INFO] hadoop-yarn-applications ... SKIPPED > [INFO] hadoop-yarn-applications-distributedshell .. SKIPPED > [INFO] hadoop-yarn-applications-unmanaged-am-launcher . SKIPPED > [INFO] hadoop-yarn-site ... SKIPPED > [INFO] hadoop-yarn-project SKIPPED > [INFO] hadoop-mapreduce-client SKIPPED > [INFO] hadoop-mapreduce-client-core ... SKIPPED > [INFO] hadoop-mapreduce-client-common . SKIPPED > [INFO] hadoop-mapreduce-client-shuffle SKIPPED > [INFO] hadoop-mapreduce-client-app SKIPPED > [INFO] hadoop
[jira] [Updated] (HADOOP-10968) hadoop common fails to detect java_libarch on ppc64le
[ https://issues.apache.org/jira/browse/HADOOP-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinar Valeev updated HADOOP-10968: -- Attachment: 0001-Set-java_libarch-for-ppc64le-1.patch Improved version, check first if ppc64le is used. Note: IBM Java is using it. > hadoop common fails to detect java_libarch on ppc64le > - > > Key: HADOOP-10968 > URL: https://issues.apache.org/jira/browse/HADOOP-10968 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.3.0 >Reporter: Dinar Valeev > Attachments: 0001-Set-java_libarch-for-ppc64le-1.patch > > > [INFO] > [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-common --- > [INFO] Executing tasks > main: > [exec] -- The C compiler identification is GNU 4.8.3 > [exec] -- The CXX compiler identification is GNU 4.8.3 > [exec] -- Check for working C compiler: /usr/bin/cc > [exec] -- Check for working C compiler: /usr/bin/cc -- works > [exec] -- Detecting C compiler ABI info > [exec] -- Detecting C compiler ABI info - done > [exec] -- Check for working CXX compiler: /usr/bin/c++ > [exec] -- Check for working CXX compiler: /usr/bin/c++ -- works > [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND > [exec] > JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include, > JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux > [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE): > [exec] Failed to find a viable JVM installation under JAVA_HOME. > [exec] Call Stack (most recent call first): > [exec] CMakeLists.txt:24 (include) > [exec] > [exec] > [exec] -- Detecting CXX compiler ABI info > [exec] -- Detecting CXX compiler ABI info - done > [exec] -- Configuring incomplete, errors occurred! > [exec] See also > "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log". > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ 10.680 > s] > [INFO] Apache Hadoop Project POM .. SUCCESS [ 0.716 > s] > [INFO] Apache Hadoop Annotations .. SUCCESS [ 3.270 > s] > [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.274 > s] > [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 1.819 > s] > [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.284 > s] > [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 2.863 > s] > [INFO] Apache Hadoop Auth . SUCCESS [ 4.032 > s] > [INFO] Apache Hadoop Auth Examples SUCCESS [ 2.475 > s] > [INFO] Apache Hadoop Common ... FAILURE [ 10.458 > s] > [INFO] Apache Hadoop NFS .. SKIPPED > [INFO] Apache Hadoop Common Project ... SKIPPED > [INFO] Apache Hadoop HDFS . SKIPPED > [INFO] Apache Hadoop HttpFS ... SKIPPED > [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED > [INFO] Apache Hadoop HDFS-NFS . SKIPPED > [INFO] Apache Hadoop HDFS Project . SKIPPED > [INFO] hadoop-yarn SKIPPED > [INFO] hadoop-yarn-api SKIPPED > [INFO] hadoop-yarn-common . SKIPPED > [INFO] hadoop-yarn-server . SKIPPED > [INFO] hadoop-yarn-server-common .. SKIPPED > [INFO] hadoop-yarn-server-nodemanager . SKIPPED > [INFO] hadoop-yarn-server-web-proxy ... SKIPPED > [INFO] hadoop-yarn-server-resourcemanager . SKIPPED > [INFO] hadoop-yarn-server-tests ... SKIPPED > [INFO] hadoop-yarn-client . SKIPPED > [INFO] hadoop-yarn-applications ... SKIPPED > [INFO] hadoop-yarn-applications-distributedshell .. SKIPPED > [INFO] hadoop-yarn-applications-unmanaged-am-launcher . SKIPPED > [INFO] hadoop-yarn-site ... SKIPPED > [INFO] hadoop-yarn-project SKIPPED > [INFO] hadoop-mapreduce-client SKIPPED > [INFO] hadoop-mapreduce-client-core ... SKIPPED > [INFO] hadoop-mapreduce-client-common . SKIPPED > [INFO] hadoop-mapreduce-client-shuffle SKIPPED > [INFO] hado
