[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125423#comment-14125423 ] Akira AJISAKA commented on HADOOP-8989: --- Thanks [~jonallen] for the update. LGTM, +1(non-binding). The test failure is not related to the patch. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11073) Credential Provider related Unit Tests Failure on Windows
[
https://issues.apache.org/jira/browse/HADOOP-11073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-11073:
---
Resolution: Fixed
Fix Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I committed this to trunk and branch-2. Xiaoyu, thank you for providing this
patch.
Credential Provider related Unit Tests Failure on Windows
-
Key: HADOOP-11073
URL: https://issues.apache.org/jira/browse/HADOOP-11073
Project: Hadoop Common
Issue Type: Test
Components: security
Affects Versions: 2.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Fix For: 2.6.0
Attachments: HADOOP-11073.0.patch, HADOOP-11073.1.patch
Reported by: Xiaomara and investigated by [~cnauroth].
The credential provider related unit tests failed on Windows. The tests try
to set up a URI by taking the build test directory and concatenating it with
other strings containing the rest of the URI format, i.e.:
{code}
public void testFactory() throws Exception {
Configuration conf = new Configuration();
conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
UserProvider.SCHEME_NAME + :///, +
JavaKeyStoreProvider.SCHEME_NAME + ://file + tmpDir +
/test.jks);
{code}
This logic is incorrect on Windows, because the file path separator will be
'\', which violates URI syntax. Forward slash is not permitted.
The proper fix is to always do path/URI construction through the
org.apache.hadoop.fs.Path class, specifically using the constructors that
take explicit parent and child arguments.
The affected unit tests are:
{code}
* TestKeyProviderFactory
* TestLdapGroupsMapping
* TestCredentialProviderFactory
* KeyStoreTestUtil
* TestSSLFactory
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125735#comment-14125735
]
Todd Lipcon commented on HADOOP-11064:
--
[~ste...@apache.org] -- hmm, I'm not 100% following here. Are you suggesting
that we are supposed to support mixed-version installations? I'd always assumed
that, if someone is running hadoop 2.4 client, then they would _only_ have
hadoop 2.4 artifacts on their path (classpath and java.library.path alike).
Here you're saying that the user has Hadoop 2.4 jars with Hadoop 2.6 SOs on the
same classpath.
In the same way that we don't expect HDFS 2.6 to work with a Common 2.4 JAR, I
don't think this is a supported scenario. Is this scenario listed in any of our
compatibility documentation?
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125742#comment-14125742
]
Todd Lipcon commented on HADOOP-11064:
--
sorry, hit send too early:
Doing a fix to maintain the old private API is certainly an easy option like
you suggested, but it would be nice to have this compatibility guarantee
written down somewhere if we feel it is something we want to maintain. I'm -0
on guaranteeing support for mixed version classpaths, since the testing matrix
becomes quite large, and it means that *internal* APIs (these methods are
marked InterfaceAudience.Private) now need attention with regard to
compatibility. What distinguishes the shared library case from the mixed
hdfs/common case mentioned above?
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[ https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-11071: Attachment: HADOOP-11071.patch new patch, previous patch was not correct as the draining was done in the wrong abstraction and it was not covering the server side correctly. Also added testcase that verifies draining on both client and server side. And, I've verified with HDFS-7006 as well. KMSClientProvider should drain the local generated EEK cache on key rollover Key: HADOOP-11071 URL: https://issues.apache.org/jira/browse/HADOOP-11071 Project: Hadoop Common Issue Type: Test Components: security Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Minor Attachments: HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch This is for formal correctness and to enable HDFS EZ to verify a rollover when testing with KMS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[ https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-11071: Attachment: HADOOP-11071.patch There was some javadoc mis-formatting in the last patch, updating. KMSClientProvider should drain the local generated EEK cache on key rollover Key: HADOOP-11071 URL: https://issues.apache.org/jira/browse/HADOOP-11071 Project: Hadoop Common Issue Type: Test Components: security Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Minor Attachments: HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch This is for formal correctness and to enable HDFS EZ to verify a rollover when testing with KMS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125799#comment-14125799
]
Nick Dimiduk commented on HADOOP-11064:
---
Backward compatibility within a major version is usually expected from a user
perspective.
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[
https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125822#comment-14125822
]
Hadoop QA commented on HADOOP-11071:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667193/HADOOP-11071.patch
against trunk revision 302d9a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms:
org.apache.hadoop.crypto.key.kms.server.TestKMS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4669//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4669//console
This message is automatically generated.
KMSClientProvider should drain the local generated EEK cache on key rollover
Key: HADOOP-11071
URL: https://issues.apache.org/jira/browse/HADOOP-11071
Project: Hadoop Common
Issue Type: Test
Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Minor
Attachments: HADOOP-11071.patch, HADOOP-11071.patch,
HADOOP-11071.patch, HADOOP-11071.patch
This is for formal correctness and to enable HDFS EZ to verify a rollover
when testing with KMS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[
https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125826#comment-14125826
]
Andrew Wang commented on HADOOP-11071:
--
I took a closer look at this, with synchronization in mind. The hypothetical
concern here would be that an async filler thread might be putting an old key
when the drain happens. Additionally, the Oracle docs say that
BlockingQueue#addAll isn't atomic, so {{keyQueue.addAll}} in the
{{fillQueueForKey}} implementations is suspect. We likely need to add some
locking.
Regarding testing, is there a way to test that before rolling, there's a cached
key on both the client and server side, and then after, there isn't on either?
Thanks Tucu.
KMSClientProvider should drain the local generated EEK cache on key rollover
Key: HADOOP-11071
URL: https://issues.apache.org/jira/browse/HADOOP-11071
Project: Hadoop Common
Issue Type: Test
Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Minor
Attachments: HADOOP-11071.patch, HADOOP-11071.patch,
HADOOP-11071.patch, HADOOP-11071.patch
This is for formal correctness and to enable HDFS EZ to verify a rollover
when testing with KMS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[ https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125860#comment-14125860 ] Alejandro Abdelnur commented on HADOOP-11071: - Regarding addAll not being atomic, that is not an issue since atomicity is not a concern and we are using threadsafe data structures. So I think we are good there. Regarding testing, the added test, using KMS, verifies the client server caches are drained because of a rollover, getting a new EEK produces an EEK using the new key version. Note that under normal circumstances this is not the typical scenario (the client rolling the key also consuming generated keys), though this is exercised by HDFS-7006. In a typical usecase, a key shell invocation will roll a key triggering the KMS to flush its server side pre-generated EEKs, any client (ie HDFS NN) that has client side pre-generated keys will continue consuming the pre-generated EEKs with the prior key version. This means that there is a maximum number of EEKs with the prior key version that can used after a rollover. If this a is a concern, client side pre-generated keys may be disabled in the NN with a performance trade off. IMO, it is reasonable to assume that at most X (X being the client side pre-generated EEK cache) EEK with a prior key version will be used. test failure is due to too many open files. KMSClientProvider should drain the local generated EEK cache on key rollover Key: HADOOP-11071 URL: https://issues.apache.org/jira/browse/HADOOP-11071 Project: Hadoop Common Issue Type: Test Components: security Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Minor Attachments: HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch This is for formal correctness and to enable HDFS EZ to verify a rollover when testing with KMS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[
https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125866#comment-14125866
]
Hadoop QA commented on HADOOP-11071:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667199/HADOOP-11071.patch
against trunk revision 302d9a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms:
org.apache.hadoop.ha.TestZKFailoverControllerStress
org.apache.hadoop.crypto.key.kms.server.TestKMS
The following test timeouts occurred in
hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms:
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4670//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4670//console
This message is automatically generated.
KMSClientProvider should drain the local generated EEK cache on key rollover
Key: HADOOP-11071
URL: https://issues.apache.org/jira/browse/HADOOP-11071
Project: Hadoop Common
Issue Type: Test
Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Minor
Attachments: HADOOP-11071.patch, HADOOP-11071.patch,
HADOOP-11071.patch, HADOOP-11071.patch
This is for formal correctness and to enable HDFS EZ to verify a rollover
when testing with KMS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[ https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125875#comment-14125875 ] Andrew Wang commented on HADOOP-11071: -- +1, yea that all makes sense. I was thinking about HDFS-6971, since I'd like to have a way of ensuring that no old keys are being used on the NN. We can take care of the concurrency stuff there. KMSClientProvider should drain the local generated EEK cache on key rollover Key: HADOOP-11071 URL: https://issues.apache.org/jira/browse/HADOOP-11071 Project: Hadoop Common Issue Type: Test Components: security Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Minor Attachments: HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch This is for formal correctness and to enable HDFS EZ to verify a rollover when testing with KMS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11071) KMSClientProvider should drain the local generated EEK cache on key rollover
[ https://issues.apache.org/jira/browse/HADOOP-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-11071: Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk and branch-2. KMSClientProvider should drain the local generated EEK cache on key rollover Key: HADOOP-11071 URL: https://issues.apache.org/jira/browse/HADOOP-11071 Project: Hadoop Common Issue Type: Test Components: security Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Minor Fix For: 2.6.0 Attachments: HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch, HADOOP-11071.patch This is for formal correctness and to enable HDFS EZ to verify a rollover when testing with KMS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125965#comment-14125965
]
Steve Loughran commented on HADOOP-11064:
-
I'm saying if someone untars the hbase binaries and tries to use them in a
hadoop 2.6 cluster, they get to see a stack trace. HBase includes all its
JARs, which are 100% in sync, it's just that nobody bundles the native libs too.
I dont care about cross-JAR compatibility, but the changes in the .lib/.so code
stop me running any hadoop 2.4 code if the 2.6 libs are on their LIBPATH. That
includes standalone applications which would otherwise run happily without any
native JARs.
IMO those native bits of code are something we need to keep stable, even though
they are never intended for direct public use. We should also have the java
clients pick up version problems (2.6 code loading 2.4 lib)
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125968#comment-14125968
]
Allen Wittenauer commented on HADOOP-11064:
---
If nothing else, we should really properly version them instead of making up
wackadoodle numbers. (libhadoop.so has been 1.0.0 for how many releases now?
Probably since it was introduced!)
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11035) distcp on mr1(branch-1) fails with NPE using a short relative source path.
[
https://issues.apache.org/jira/browse/HADOOP-11035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125977#comment-14125977
]
Andrew Wang commented on HADOOP-11035:
--
+1 LGTM, will commit this shortly.
distcp on mr1(branch-1) fails with NPE using a short relative source path.
--
Key: HADOOP-11035
URL: https://issues.apache.org/jira/browse/HADOOP-11035
Project: Hadoop Common
Issue Type: Bug
Components: tools
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: HADOOP-11035.000.patch, HADOOP-11035.001.patch,
HADOOP-11035.002.patch
distcp on mr1(branch-1) fails with NPE using a short relative source path.
The failure is at DistCp.java, makeRelative return null at the following code:
The parameters passed to makeRelative are not same format:
root is relative path and child.getPath() is a full path.
{code}
final String dst = makeRelative(root, child.getPath());
{code}
The solution is
change root to full path to match child.getPath().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125978#comment-14125978
]
Colin Patrick McCabe commented on HADOOP-11064:
---
bq. This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
I don't follow. What is to rebuild? If you put the Hadoop 2.6 jars on the
classpath and the hadoop 2.6 libhadoop.so on the classpath, your application
should work.
bq. Backward compatibility within a major version is usually expected from a
user perspective.
As Todd mentioned, we don't support mixing Hadoop 2.4 and Hadoop 2.6 jars on
the same CLASSPATH. Similarly, we don't support mixing a libhadoop.so from one
version with a libhadoop.so from another version. Backwards compatibility
refers to compatibility with the public APIs, not compatibility with internal,
non-public APIs.
There's been a few of these jiras recently where people seem to want to put
multiple versions of libhadoop.so on their classpath and expect it to work (I
guess it would be hadoop.dll on Windows). I don't understand the motivation
for doing this-- can't the launcher scripts for Windows simply set up the
CLASSPATH and LD_LIBRARY_PATH appropriately for the version being launched?
What am I missing?
If it's absolutely, positively necessary to support throwing multiple versions
of the libhadoop library on the classpath at once, we could inject the version
into the library name / version info.
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11035) distcp on mr1(branch-1) fails with NPE using a short relative source path.
[
https://issues.apache.org/jira/browse/HADOOP-11035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-11035:
-
Resolution: Fixed
Fix Version/s: 1.3.0
Status: Resolved (was: Patch Available)
Committed to branch-1, thanks Zhihai for the patch, Yongjun for reviewing.
distcp on mr1(branch-1) fails with NPE using a short relative source path.
--
Key: HADOOP-11035
URL: https://issues.apache.org/jira/browse/HADOOP-11035
Project: Hadoop Common
Issue Type: Bug
Components: tools
Reporter: zhihai xu
Assignee: zhihai xu
Fix For: 1.3.0
Attachments: HADOOP-11035.000.patch, HADOOP-11035.001.patch,
HADOOP-11035.002.patch
distcp on mr1(branch-1) fails with NPE using a short relative source path.
The failure is at DistCp.java, makeRelative return null at the following code:
The parameters passed to makeRelative are not same format:
root is relative path and child.getPath() is a full path.
{code}
final String dst = makeRelative(root, child.getPath());
{code}
The solution is
change root to full path to match child.getPath().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11034) ViewFileSystem is missing getStatus(Path)
[ https://issues.apache.org/jira/browse/HADOOP-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Steelman updated HADOOP-11034: --- Attachment: (was: HADOOP-11034.2.patch) ViewFileSystem is missing getStatus(Path) - Key: HADOOP-11034 URL: https://issues.apache.org/jira/browse/HADOOP-11034 Project: Hadoop Common Issue Type: Bug Components: viewfs Reporter: Gary Steelman Attachments: HADOOP-11034-trunk-1.patch This patch implements ViewFileSystem#getStatus(Path), which is currently unimplemented. getStatus(Path) should return the FsStatus of the FileSystem backing the path. Currently it returns the same as getStatus(), which is a default Long.MAX_VALUE for capacity, 0 used, and Long.MAX_VALUE for remaining space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11034) ViewFileSystem is missing getStatus(Path)
[ https://issues.apache.org/jira/browse/HADOOP-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Steelman updated HADOOP-11034: --- Attachment: HADOOP-11034.2.patch Reattaching patch 2 to trigger Jenkins build. ViewFileSystem is missing getStatus(Path) - Key: HADOOP-11034 URL: https://issues.apache.org/jira/browse/HADOOP-11034 Project: Hadoop Common Issue Type: Bug Components: viewfs Reporter: Gary Steelman Attachments: HADOOP-11034-trunk-1.patch, HADOOP-11034.2.patch This patch implements ViewFileSystem#getStatus(Path), which is currently unimplemented. getStatus(Path) should return the FsStatus of the FileSystem backing the path. Currently it returns the same as getStatus(), which is a default Long.MAX_VALUE for capacity, 0 used, and Long.MAX_VALUE for remaining space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[ https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125989#comment-14125989 ] Colin Patrick McCabe commented on HADOOP-11062: --- Thanks for looking at this, Arun. I agree with Charles that we should log something when we skip. I tested with the second patch you uploaded (you might want to consider giving each patch a different name, btw), and it didn't output anything when skipping. (It does log output when it ran the test via Pnative.) Looks pretty good aside from that, though. CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used -- Key: HADOOP-11062 URL: https://issues.apache.org/jira/browse/HADOOP-11062 Project: Hadoop Common Issue Type: Bug Components: security, test Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Arun Suresh Attachments: HADOOP-11062.1.patch, HADOOP-11062.1.patch there are a few testcases, cryptocodec related that require Hadoop native code and OpenSSL. These tests should be skipped if -Pnative is not used when running the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11044) FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps
[ https://issues.apache.org/jira/browse/HADOOP-11044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125991#comment-14125991 ] Gary Steelman commented on HADOOP-11044: Hey [~sdaingade] I will take a look at this soon and see if there's anything I can help with. FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps Key: HADOOP-11044 URL: https://issues.apache.org/jira/browse/HADOOP-11044 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.5.0, 2.4.1 Reporter: Swapnil Daingade Priority: Minor Attachments: 11044.patch4 The org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData class defines readOps, largeReadOps, writeOps as int. Also the The org.apache.hadoop.fs.FileSystem.Statistics class has methods like getReadOps(), getLargeReadOps() and getWriteOps() that return int. These int values can overflow if the exceed 2^31-1 showing negative values. It would be nice if these can be changed to long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11032) Replace use of Guava Stopwatch with Apache StopWatch
[ https://issues.apache.org/jira/browse/HADOOP-11032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126002#comment-14126002 ] Gary Steelman commented on HADOOP-11032: Thanks [~ozawa]. The test failures don't appear to be related to this patch. For fixing the exception thrown when StopWatch.stop() is called twice consecutively, ideally the fix is to simply not call stop() twice consecutively. I meant, however, we could submit a patch to the commons project to change the behavior of stop() to Log.warn(StopWatch is already in stopped state); or be a no-op instead of throwing an Exception. Replace use of Guava Stopwatch with Apache StopWatch Key: HADOOP-11032 URL: https://issues.apache.org/jira/browse/HADOOP-11032 Project: Hadoop Common Issue Type: Improvement Reporter: Gary Steelman Assignee: Tsuyoshi OZAWA Attachments: HADOOP-11032.1.patch, HADOOP-11032.2.patch, HADOOP-11032.3.patch, HADOOP-11032.3.patch This patch reduces Hadoop's dependency on an old version of guava. Stopwatch.elapsedMillis() isn't part of guava past v16 and the tools I'm working on use v17. To remedy this and also reduce Hadoop's reliance on old versions of guava, we can use the Apache StopWatch (org.apache.commons.lang.time.StopWatch) which provides nearly equivalent functionality. apache.commons.lang is already a dependency for Hadoop so this will not introduce new dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11038) Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
[
https://issues.apache.org/jira/browse/HADOOP-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Gupta updated HADOOP-11038:
--
Attachment: (was: HADOOP-11038.2.patch)
Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
-
Key: HADOOP-11038
URL: https://issues.apache.org/jira/browse/HADOOP-11038
Project: Hadoop Common
Issue Type: New Feature
Components: fs, fs/s3
Reporter: Vishal Gupta
Currently hdfs dfs -lsr s3://... supports acess-keys/secret-keys only as
the way to authenticate to s3. This should support AWS-roles also because of
the following reasons :
1) AWS-roles is a AWS best-practice and is highly recommended by AWS
themselves.
2) This helps in cross-AWS-account integration also. An AWS-account-holder
can provide another AWS-account-holder a cross-account-AWS-role to perform
operations over his S3-buckets.
The current syntax is hdfs dfs is :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= -ls
s3n://.../
This should change to :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey=
-Dfs.s3n.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3n.awsExternalId= -ls s3n://.../
Extending the use-case a little further, for a client AWS-account to
integrate with multiple different AWS-accounts, configuration for s3-bucket
to role-to-be-assumed mapping ( which will override the master-role ) should
be provided :
hdfs dfs -Dfs.s3.awsAccessKeyId= -Dfs.s3.awsSecretAccessKey=
-Dfs.s3.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3.awsBucketToRoleMapping={\bucket1\: {
\roleName\:\arn:aws:iam:::role/role1\, \externalId\:\\}} -ls
s3://.../
Since, AWS treats a cross-account-AWS-role the same as an AWS-role within a
AWS-account, the above flows remain same for a role within a AWS-account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11038) Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
[
https://issues.apache.org/jira/browse/HADOOP-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Gupta updated HADOOP-11038:
--
Attachment: (was: HADOOP-11038.1.patch)
Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
-
Key: HADOOP-11038
URL: https://issues.apache.org/jira/browse/HADOOP-11038
Project: Hadoop Common
Issue Type: New Feature
Components: fs, fs/s3
Reporter: Vishal Gupta
Attachments: HADOOP-11038.2.patch
Currently hdfs dfs -lsr s3://... supports acess-keys/secret-keys only as
the way to authenticate to s3. This should support AWS-roles also because of
the following reasons :
1) AWS-roles is a AWS best-practice and is highly recommended by AWS
themselves.
2) This helps in cross-AWS-account integration also. An AWS-account-holder
can provide another AWS-account-holder a cross-account-AWS-role to perform
operations over his S3-buckets.
The current syntax is hdfs dfs is :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= -ls
s3n://.../
This should change to :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey=
-Dfs.s3n.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3n.awsExternalId= -ls s3n://.../
Extending the use-case a little further, for a client AWS-account to
integrate with multiple different AWS-accounts, configuration for s3-bucket
to role-to-be-assumed mapping ( which will override the master-role ) should
be provided :
hdfs dfs -Dfs.s3.awsAccessKeyId= -Dfs.s3.awsSecretAccessKey=
-Dfs.s3.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3.awsBucketToRoleMapping={\bucket1\: {
\roleName\:\arn:aws:iam:::role/role1\, \externalId\:\\}} -ls
s3://.../
Since, AWS treats a cross-account-AWS-role the same as an AWS-role within a
AWS-account, the above flows remain same for a role within a AWS-account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11038) Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
[
https://issues.apache.org/jira/browse/HADOOP-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Gupta updated HADOOP-11038:
--
Attachment: HADOOP-11038.2.patch
HADOOP-11038.1.patch
Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
-
Key: HADOOP-11038
URL: https://issues.apache.org/jira/browse/HADOOP-11038
Project: Hadoop Common
Issue Type: New Feature
Components: fs, fs/s3
Reporter: Vishal Gupta
Attachments: HADOOP-11038.1.patch, HADOOP-11038.2.patch
Currently hdfs dfs -lsr s3://... supports acess-keys/secret-keys only as
the way to authenticate to s3. This should support AWS-roles also because of
the following reasons :
1) AWS-roles is a AWS best-practice and is highly recommended by AWS
themselves.
2) This helps in cross-AWS-account integration also. An AWS-account-holder
can provide another AWS-account-holder a cross-account-AWS-role to perform
operations over his S3-buckets.
The current syntax is hdfs dfs is :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= -ls
s3n://.../
This should change to :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey=
-Dfs.s3n.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3n.awsExternalId= -ls s3n://.../
Extending the use-case a little further, for a client AWS-account to
integrate with multiple different AWS-accounts, configuration for s3-bucket
to role-to-be-assumed mapping ( which will override the master-role ) should
be provided :
hdfs dfs -Dfs.s3.awsAccessKeyId= -Dfs.s3.awsSecretAccessKey=
-Dfs.s3.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3.awsBucketToRoleMapping={\bucket1\: {
\roleName\:\arn:aws:iam:::role/role1\, \externalId\:\\}} -ls
s3://.../
Since, AWS treats a cross-account-AWS-role the same as an AWS-role within a
AWS-account, the above flows remain same for a role within a AWS-account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11074) Move s3-related FS connector code to hadoop-aws
David S. Wang created HADOOP-11074: -- Summary: Move s3-related FS connector code to hadoop-aws Key: HADOOP-11074 URL: https://issues.apache.org/jira/browse/HADOOP-11074 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0 Reporter: David S. Wang Assignee: David S. Wang Now that hadoop-aws has been created, we should actually move the relevant code into that module, similar to what was done with hadoop-openstack, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11038) Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
[
https://issues.apache.org/jira/browse/HADOOP-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126041#comment-14126041
]
Hadoop QA commented on HADOOP-11038:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667232/HADOOP-11038.2.patch
against trunk revision d989ac0.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4672//console
This message is automatically generated.
Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
-
Key: HADOOP-11038
URL: https://issues.apache.org/jira/browse/HADOOP-11038
Project: Hadoop Common
Issue Type: New Feature
Components: fs, fs/s3
Reporter: Vishal Gupta
Attachments: HADOOP-11038.1.patch, HADOOP-11038.2.patch
Currently hdfs dfs -lsr s3://... supports acess-keys/secret-keys only as
the way to authenticate to s3. This should support AWS-roles also because of
the following reasons :
1) AWS-roles is a AWS best-practice and is highly recommended by AWS
themselves.
2) This helps in cross-AWS-account integration also. An AWS-account-holder
can provide another AWS-account-holder a cross-account-AWS-role to perform
operations over his S3-buckets.
The current syntax is hdfs dfs is :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= -ls
s3n://.../
This should change to :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey=
-Dfs.s3n.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3n.awsExternalId= -ls s3n://.../
Extending the use-case a little further, for a client AWS-account to
integrate with multiple different AWS-accounts, configuration for s3-bucket
to role-to-be-assumed mapping ( which will override the master-role ) should
be provided :
hdfs dfs -Dfs.s3.awsAccessKeyId= -Dfs.s3.awsSecretAccessKey=
-Dfs.s3.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3.awsBucketToRoleMapping={\bucket1\: {
\roleName\:\arn:aws:iam:::role/role1\, \externalId\:\\}} -ls
s3://.../
Since, AWS treats a cross-account-AWS-role the same as an AWS-role within a
AWS-account, the above flows remain same for a role within a AWS-account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11038) Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
[
https://issues.apache.org/jira/browse/HADOOP-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126066#comment-14126066
]
Vishal Gupta commented on HADOOP-11038:
---
Please find the updated patches attached.
1) HADOOP-11038.1.patch needs to applied to hadoop-tree.
2) HADOOP-11038.2.patch needs to applied to 0.9.0 version of jets3t. ( the code
changes can also be found at https://github.com/guptavishal/jets3t-aws-roles ).
If the patched version of jets3t in (2) is included in the hadoop-tree, the
options for AWS-roles start getting supported, else the behavior remains the
same as before.
Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
-
Key: HADOOP-11038
URL: https://issues.apache.org/jira/browse/HADOOP-11038
Project: Hadoop Common
Issue Type: New Feature
Components: fs, fs/s3
Reporter: Vishal Gupta
Attachments: HADOOP-11038.1.patch, HADOOP-11038.2.patch
Currently hdfs dfs -lsr s3://... supports acess-keys/secret-keys only as
the way to authenticate to s3. This should support AWS-roles also because of
the following reasons :
1) AWS-roles is a AWS best-practice and is highly recommended by AWS
themselves.
2) This helps in cross-AWS-account integration also. An AWS-account-holder
can provide another AWS-account-holder a cross-account-AWS-role to perform
operations over his S3-buckets.
The current syntax is hdfs dfs is :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= -ls
s3n://.../
This should change to :
hdfs dfs -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey=
-Dfs.s3n.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3n.awsExternalId= -ls s3n://.../
Extending the use-case a little further, for a client AWS-account to
integrate with multiple different AWS-accounts, configuration for s3-bucket
to role-to-be-assumed mapping ( which will override the master-role ) should
be provided :
hdfs dfs -Dfs.s3.awsAccessKeyId= -Dfs.s3.awsSecretAccessKey=
-Dfs.s3.awsRoleToBeAssumed=arn:aws:iam:::role/
-Dfs.s3.awsBucketToRoleMapping={\bucket1\: {
\roleName\:\arn:aws:iam:::role/role1\, \externalId\:\\}} -ls
s3://.../
Since, AWS treats a cross-account-AWS-role the same as an AWS-role within a
AWS-account, the above flows remain same for a role within a AWS-account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao reassigned HADOOP-11057:
---
Assignee: Xiaoyu Yao
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11075) hadoop-kms is not being published to maven
Anthony Young-Garner created HADOOP-11075: - Summary: hadoop-kms is not being published to maven Key: HADOOP-11075 URL: https://issues.apache.org/jira/browse/HADOOP-11075 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Anthony Young-Garner Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11075) hadoop-kms is not being published to maven
[ https://issues.apache.org/jira/browse/HADOOP-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Young-Garner updated HADOOP-11075: -- Attachment: 0001-Exposed-hadoop-kms-classes-on-hadoop-KMS-POM.patch hadoop-kms is not being published to maven -- Key: HADOOP-11075 URL: https://issues.apache.org/jira/browse/HADOOP-11075 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Anthony Young-Garner Priority: Minor Attachments: 0001-Exposed-hadoop-kms-classes-on-hadoop-KMS-POM.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11076) SbNN Web UI shows #Under replicated blocks and #pending deletion blocks
Juan Yu created HADOOP-11076: Summary: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks Key: HADOOP-11076 URL: https://issues.apache.org/jira/browse/HADOOP-11076 Project: Hadoop Common Issue Type: Bug Reporter: Juan Yu I believe that's an regression of HDFS-5333 According to HDFS-2901 and HDFS-6178 The Standby Namenode doesn't compute replication queues, we shouldn't show under-replicated/missing blocks or corrupt files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11076) SbNN Web UI shows #Under replicated blocks and #pending deletion blocks
[ https://issues.apache.org/jira/browse/HADOOP-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Yu updated HADOOP-11076: - Component/s: ha SbNN Web UI shows #Under replicated blocks and #pending deletion blocks --- Key: HADOOP-11076 URL: https://issues.apache.org/jira/browse/HADOOP-11076 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Juan Yu I believe that's an regression of HDFS-5333 According to HDFS-2901 and HDFS-6178 The Standby Namenode doesn't compute replication queues, we shouldn't show under-replicated/missing blocks or corrupt files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-11057 started by Xiaoyu Yao.
---
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11009) Add Timestamp Preservation to DistCp
[ https://issues.apache.org/jira/browse/HADOOP-11009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126195#comment-14126195 ] Gary Steelman commented on HADOOP-11009: Great! Looks like all the tests passed. If someone else can take a look at this too I would appreciate it. Add Timestamp Preservation to DistCp Key: HADOOP-11009 URL: https://issues.apache.org/jira/browse/HADOOP-11009 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Affects Versions: 2.4.0 Reporter: Gary Steelman Attachments: HADOOP-11009.1.patch, HADOOP-11009.2.patch, HADOOP-11009.3.patch Currently access and modification times are not preserved on files copied using DistCp. This patch adds an option to DistCp for timestamp preservation. The patch ready, but I understand there is a Contributor form I need to sign before I can upload it. Can someone point me in the right direction for this form? Thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10903) Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest.
[ https://issues.apache.org/jira/browse/HADOOP-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10903: --- Release Note: The hadoop classpath command has been enhanced to support options for automatic expansion of wildcards in classpath elements and writing the classpath to a jar file manifest. These options make it easier to construct a correct classpath for libhdfs applications. Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest. -- Key: HADOOP-10903 URL: https://issues.apache.org/jira/browse/HADOOP-10903 Project: Hadoop Common Issue Type: Improvement Components: scripts, util Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.6.0 Attachments: HADOOP-10903.1.patch The hadoop classpath shell command currently prints the classpath variable established by the shell scripts and then exits. A few enhancements to this command would be desirable to support a few other use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126207#comment-14126207
]
Todd Lipcon commented on HADOOP-11064:
--
bq. If nothing else, we should really properly version them instead of making
up wackadoodle numbers. (libhadoop.so has been 1.0.0 for how many releases now?
Probably since it was introduced!)
Agreed, but unfortunately afaik Java doesn't provide a way to specify a
particular so version dependency. I asked this on Quora a few years ago:
http://www.quora.com/Is-there-a-way-to-force-Java-to-load-a-particular-soversion-of-a-JNI-dependency
and unfortunately got no real answers.
So, does this mean we need to always keep binary compatibility of the
internal-facing libhadoop.so? Could we change hbase to pick up the Hadoop
libraries from HADOOP_HOME instead of bundling them? It seems like it should
either (a) bundle everything, including the native code, or (b) bundle nothing,
and load everything from HADOOP_HOME. What's causing a problem is that it's
using bundled jars with system-located native code.
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11029) FileSystem#Statistics uses volatile variables that must be updated on write or read calls.
[
https://issues.apache.org/jira/browse/HADOOP-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11029:
-
Attachment: fsdatainputstream.png
FileSystem#Statistics uses volatile variables that must be updated on write
or read calls.
--
Key: HADOOP-11029
URL: https://issues.apache.org/jira/browse/HADOOP-11029
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.5.0, 2.6.0
Reporter: Gopal V
Attachments: local-fs-locking.png
This code is there in the hot-path of IFile writer via RawLocalFileSystem.
!local-fs-locking.png!
From a preliminary glance, the lock prefix calls are coming from a
threadlocal.get() within FileSystem.Statistics
{code}
/**
* Get or create the thread-local data associated with the current thread.
*/
private StatisticsData getThreadData() {
StatisticsData data = threadData.get();
if (data == null) {
data = new StatisticsData(
new WeakReferenceThread(Thread.currentThread()));
threadData.set(data);
synchronized(this) {
if (allData == null) {
allData = new LinkedListStatisticsData();
}
allData.add(data);
}
}
return data;
}
/**
* Increment the bytes read in the statistics
* @param newBytes the additional bytes read
*/
public void incrementBytesRead(long newBytes) {
getThreadData().bytesRead += newBytes;
}
{code}
This is incredibly inefficient when used from FSDataOutputStream
{code}
public void write(int b) throws IOException {
out.write(b);
position++;
if (statistics != null) {
statistics.incrementBytesWritten(1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11029) FileSystem#Statistics uses volatile variables that must be updated on write or read calls.
[
https://issues.apache.org/jira/browse/HADOOP-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11029:
-
Attachment: (was: fsdatainputstream.png)
FileSystem#Statistics uses volatile variables that must be updated on write
or read calls.
--
Key: HADOOP-11029
URL: https://issues.apache.org/jira/browse/HADOOP-11029
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.5.0, 2.6.0
Reporter: Gopal V
Attachments: local-fs-locking.png
This code is there in the hot-path of IFile writer via RawLocalFileSystem.
!local-fs-locking.png!
From a preliminary glance, the lock prefix calls are coming from a
threadlocal.get() within FileSystem.Statistics
{code}
/**
* Get or create the thread-local data associated with the current thread.
*/
private StatisticsData getThreadData() {
StatisticsData data = threadData.get();
if (data == null) {
data = new StatisticsData(
new WeakReferenceThread(Thread.currentThread()));
threadData.set(data);
synchronized(this) {
if (allData == null) {
allData = new LinkedListStatisticsData();
}
allData.add(data);
}
}
return data;
}
/**
* Increment the bytes read in the statistics
* @param newBytes the additional bytes read
*/
public void incrementBytesRead(long newBytes) {
getThreadData().bytesRead += newBytes;
}
{code}
This is incredibly inefficient when used from FSDataOutputStream
{code}
public void write(int b) throws IOException {
out.write(b);
position++;
if (statistics != null) {
statistics.incrementBytesWritten(1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11029) FileSystem#Statistics uses volatile variables that must be updated on write or read calls.
[
https://issues.apache.org/jira/browse/HADOOP-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126228#comment-14126228
]
Gopal V commented on HADOOP-11029:
--
bq. Can you put some numbers behind this? If this is only an issue when you are
writing a byte at a time, I'd be inclined to close this, since, as I mentioned,
byte-at-a-time is a well-known anti-pattern with Java streams.
When running a wordcount example with a Snappy compressed stream (64kb
buffers), I find that 57% of cpu samples are split between HADOOP-10694,
HADOOP-10681, HADOOP-11029.
!fsdatainputstream.png!
This perf sample is using the write(byte[], int, int) version.
FileSystem#Statistics uses volatile variables that must be updated on write
or read calls.
--
Key: HADOOP-11029
URL: https://issues.apache.org/jira/browse/HADOOP-11029
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.5.0, 2.6.0
Reporter: Gopal V
Attachments: local-fs-locking.png
This code is there in the hot-path of IFile writer via RawLocalFileSystem.
!local-fs-locking.png!
From a preliminary glance, the lock prefix calls are coming from a
threadlocal.get() within FileSystem.Statistics
{code}
/**
* Get or create the thread-local data associated with the current thread.
*/
private StatisticsData getThreadData() {
StatisticsData data = threadData.get();
if (data == null) {
data = new StatisticsData(
new WeakReferenceThread(Thread.currentThread()));
threadData.set(data);
synchronized(this) {
if (allData == null) {
allData = new LinkedListStatisticsData();
}
allData.add(data);
}
}
return data;
}
/**
* Increment the bytes read in the statistics
* @param newBytes the additional bytes read
*/
public void incrementBytesRead(long newBytes) {
getThreadData().bytesRead += newBytes;
}
{code}
This is incredibly inefficient when used from FSDataOutputStream
{code}
public void write(int b) throws IOException {
out.write(b);
position++;
if (statistics != null) {
statistics.incrementBytesWritten(1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11029) FileSystem#Statistics uses volatile variables that must be updated on write or read calls.
[
https://issues.apache.org/jira/browse/HADOOP-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11029:
-
Attachment: fsdatainputstream.png
FileSystem#Statistics uses volatile variables that must be updated on write
or read calls.
--
Key: HADOOP-11029
URL: https://issues.apache.org/jira/browse/HADOOP-11029
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.5.0, 2.6.0
Reporter: Gopal V
Attachments: fsdatainputstream.png, local-fs-locking.png
This code is there in the hot-path of IFile writer via RawLocalFileSystem.
!local-fs-locking.png!
From a preliminary glance, the lock prefix calls are coming from a
threadlocal.get() within FileSystem.Statistics
{code}
/**
* Get or create the thread-local data associated with the current thread.
*/
private StatisticsData getThreadData() {
StatisticsData data = threadData.get();
if (data == null) {
data = new StatisticsData(
new WeakReferenceThread(Thread.currentThread()));
threadData.set(data);
synchronized(this) {
if (allData == null) {
allData = new LinkedListStatisticsData();
}
allData.add(data);
}
}
return data;
}
/**
* Increment the bytes read in the statistics
* @param newBytes the additional bytes read
*/
public void incrementBytesRead(long newBytes) {
getThreadData().bytesRead += newBytes;
}
{code}
This is incredibly inefficient when used from FSDataOutputStream
{code}
public void write(int b) throws IOException {
out.write(b);
position++;
if (statistics != null) {
statistics.incrementBytesWritten(1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126239#comment-14126239
]
Chris Nauroth commented on HADOOP-11064:
bq. ...bundle everything, including the native code...
I think there is a reliance on the Maven jar file dependency as a reliable unit
of versioning. Applications expect that if they bundle their relevant Hadoop
jars all at the same version, then everything is going to work. Unfortunately,
it turns out that hadoop-common.jar is an incomplete artifact in terms of
Maven versioning because of the tight coupling between the jar and the native
code.
If we bundled the native libs inside hadoop-common.jar (or some new jar), and
extracted it automatically at runtime, then downstream projects could get a
completely versioned artifact through the Maven dependency. The snappy-java
project is an example of something that does this. It detects the OS and
extracts the corresponding native library at runtime. I believe hadoop-lzo has
just started doing something similar too.
Of course, this opens up a can of worms around release engineering. The
current Apache release process only builds the native components on a single
platform. I can't think of a way to make this work without a lot of
infrastructure work as a pre-requisite.
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11034) ViewFileSystem is missing getStatus(Path)
[
https://issues.apache.org/jira/browse/HADOOP-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126242#comment-14126242
]
Hadoop QA commented on HADOOP-11034:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667221/HADOOP-11034.2.patch
against trunk revision 6a84f88.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4671//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4671//console
This message is automatically generated.
ViewFileSystem is missing getStatus(Path)
-
Key: HADOOP-11034
URL: https://issues.apache.org/jira/browse/HADOOP-11034
Project: Hadoop Common
Issue Type: Bug
Components: viewfs
Reporter: Gary Steelman
Attachments: HADOOP-11034-trunk-1.patch, HADOOP-11034.2.patch
This patch implements ViewFileSystem#getStatus(Path), which is currently
unimplemented.
getStatus(Path) should return the FsStatus of the FileSystem backing the
path. Currently it returns the same as getStatus(), which is a default
Long.MAX_VALUE for capacity, 0 used, and Long.MAX_VALUE for remaining space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11075) hadoop-kms is not being published to maven
[ https://issues.apache.org/jira/browse/HADOOP-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126252#comment-14126252 ] Chris Nauroth commented on HADOOP-11075: Hi, [~anthony.young-gar...@cloudera.com]. In HADOOP-11063, I had turned on the archiveClasses option. From this patch file, it looks like your local copy didn't have that change when you made the patch. I thought the classes in hadoop-kms were only used by the webapp and therefore didn't need to be published as a separate Maven artifact, which is what attachClasses would do. Is there something else that needs to depend on these classes? If something needs to use the classes as a library, then perhaps we really need to separate them into a different module to communicate the intent more clearly and simplify the dependency declaration for those users. Thanks! hadoop-kms is not being published to maven -- Key: HADOOP-11075 URL: https://issues.apache.org/jira/browse/HADOOP-11075 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Anthony Young-Garner Priority: Minor Attachments: 0001-Exposed-hadoop-kms-classes-on-hadoop-KMS-POM.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11075) hadoop-kms is not being published to maven
[ https://issues.apache.org/jira/browse/HADOOP-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-11075: --- Assignee: Anthony Young-Garner hadoop-kms is not being published to maven -- Key: HADOOP-11075 URL: https://issues.apache.org/jira/browse/HADOOP-11075 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Anthony Young-Garner Assignee: Anthony Young-Garner Priority: Minor Attachments: 0001-Exposed-hadoop-kms-classes-on-hadoop-KMS-POM.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-11057:
Attachment: HADOOP-11057.0.patch
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
Attachments: HADOOP-11057.0.patch
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-11057:
Fix Version/s: 2.4.1
Status: Patch Available (was: In Progress)
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
Fix For: 2.4.1
Attachments: HADOOP-11057.0.patch
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126265#comment-14126265
]
Allen Wittenauer commented on HADOOP-11064:
---
bq. unfortunately afaik Java doesn't provide a way to specify a particular so
version dependency.
I keep thinking that System.load(/full/path/libname.so.1.2.3) worked. (We
should be able to figure out the full path by just processing java.library.path
manually or maybe reading hadoop.native.dir or whatever.)
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-11064:
--
Attachment: HADOOP-11064.001.patch
Here's a patch I whipped up to rename {{libhadoop.so}} to
{{libhadoop-3.0.0-SNAPSHOT.so}} (or whatever your version number may happen to
be)
This is basically not using the normal versioning scheme which Java seems to
ignore in {{System.load}}, but changing the name of the library-- an important
distinction. This allows us to avoid things like parsing {{java.library.path}}
ourselves, etc.
I think this will put the issue to rest and give people some relief from
version conflicts and API issues. I don't know if there's anything that needs
to be done on the bigtop / packaging side...
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Priority: Blocker
Attachments: HADOOP-11064.001.patch
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-11064:
--
Assignee: Colin Patrick McCabe
Status: Patch Available (was: Open)
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-11064.001.patch
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-10922) User documentation for CredentialShell
[ https://issues.apache.org/jira/browse/HADOOP-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10922: - Status: Open (was: Patch Available) User documentation for CredentialShell -- Key: HADOOP-10922 URL: https://issues.apache.org/jira/browse/HADOOP-10922 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Larry McCay Attachments: HADOOP-10922-1.patch, HADOOP-10922-2.patch The CredentialShell needs end user documentation for the website. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10922) User documentation for CredentialShell
[ https://issues.apache.org/jira/browse/HADOOP-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10922: - Attachment: HADOOP-10922-3.patch Realized that previous patch had dashes in front of the subcommands which is wrong. This patch corrects that issue. User documentation for CredentialShell -- Key: HADOOP-10922 URL: https://issues.apache.org/jira/browse/HADOOP-10922 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Larry McCay Attachments: HADOOP-10922-1.patch, HADOOP-10922-2.patch, HADOOP-10922-3.patch The CredentialShell needs end user documentation for the website. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10922) User documentation for CredentialShell
[ https://issues.apache.org/jira/browse/HADOOP-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10922: - Status: Patch Available (was: Open) User documentation for CredentialShell -- Key: HADOOP-10922 URL: https://issues.apache.org/jira/browse/HADOOP-10922 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Larry McCay Attachments: HADOOP-10922-1.patch, HADOOP-10922-2.patch, HADOOP-10922-3.patch The CredentialShell needs end user documentation for the website. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10922) User documentation for CredentialShell
[ https://issues.apache.org/jira/browse/HADOOP-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126331#comment-14126331 ] Larry McCay commented on HADOOP-10922: -- [~andrew.wang] - can you please give this latest patch a review? Thanks! User documentation for CredentialShell -- Key: HADOOP-10922 URL: https://issues.apache.org/jira/browse/HADOOP-10922 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Larry McCay Attachments: HADOOP-10922-1.patch, HADOOP-10922-2.patch, HADOOP-10922-3.patch The CredentialShell needs end user documentation for the website. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126338#comment-14126338
]
Hadoop QA commented on HADOOP-11057:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667271/HADOOP-11057.0.patch
against trunk revision d989ac0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4673//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4673//console
This message is automatically generated.
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
Fix For: 2.4.1
Attachments: HADOOP-11057.0.patch
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126345#comment-14126345
]
Chris Nauroth commented on HADOOP-11064:
[~ste...@apache.org], did you want the scope of this jira focused on
reinstating the old native function signatures in {{NativeCrc32}}? That's the
fastest path to resolving the blocker.
The longer-term policy discussion is great too, but perhaps that's better
handled separately.
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-11064.001.patch
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-11057:
Attachment: HADOOP-11057.1.patch
Minor change to align the : of winutils: with others.
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
Fix For: 2.4.1
Attachments: HADOOP-11057.0.patch, HADOOP-11057.1.patch
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11077) NPE if hosts not specified in ProxyUsers
Gregory Chanan created HADOOP-11077: --- Summary: NPE if hosts not specified in ProxyUsers Key: HADOOP-11077 URL: https://issues.apache.org/jira/browse/HADOOP-11077 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.6.0 Reporter: Gregory Chanan Assignee: Gregory Chanan When using the TokenDelegationAuthenticationFilter, I noticed if I don't specify the hosts for a user/groups proxy user and then try to authenticate, I get an NPE rather than an AuthorizationException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11077) NPE if hosts not specified in ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HADOOP-11077: Status: Patch Available (was: Open) NPE if hosts not specified in ProxyUsers Key: HADOOP-11077 URL: https://issues.apache.org/jira/browse/HADOOP-11077 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.6.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HADOOP-11077.patch When using the TokenDelegationAuthenticationFilter, I noticed if I don't specify the hosts for a user/groups proxy user and then try to authenticate, I get an NPE rather than an AuthorizationException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11077) NPE if hosts not specified in ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HADOOP-11077: Attachment: HADOOP-11077.patch Here's a simple patch against trunk including a test. NPE if hosts not specified in ProxyUsers Key: HADOOP-11077 URL: https://issues.apache.org/jira/browse/HADOOP-11077 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.6.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HADOOP-11077.patch When using the TokenDelegationAuthenticationFilter, I noticed if I don't specify the hosts for a user/groups proxy user and then try to authenticate, I get an NPE rather than an AuthorizationException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126408#comment-14126408
]
Colin Patrick McCabe commented on HADOOP-11064:
---
I'm concerned that this will set a bad precedent here that we are unable to
change libhadoop.so between versions. We certainly have done this a lot in the
past, and it's important that we keep this flexibility. Since we have some
time before 2.6, shouldn't we consider the library renaming solution?
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-11064.001.patch
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126415#comment-14126415
]
Hadoop QA commented on HADOOP-11064:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667284/HADOOP-11064.001.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4674//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4674//console
This message is automatically generated.
UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
method changes
--
Key: HADOOP-11064
URL: https://issues.apache.org/jira/browse/HADOOP-11064
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
2.4 JARs
Reporter: Steve Loughran
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-11064.001.patch
The private native method names and signatures in {{NativeCrc32}} were
changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
link errors when they try to perform checksums.
This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10922) User documentation for CredentialShell
[
https://issues.apache.org/jira/browse/HADOOP-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126416#comment-14126416
]
Hadoop QA commented on HADOOP-10922:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667285/HADOOP-10922-3.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4675//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4675//console
This message is automatically generated.
User documentation for CredentialShell
--
Key: HADOOP-10922
URL: https://issues.apache.org/jira/browse/HADOOP-10922
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Andrew Wang
Assignee: Larry McCay
Attachments: HADOOP-10922-1.patch, HADOOP-10922-2.patch,
HADOOP-10922-3.patch
The CredentialShell needs end user documentation for the website.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11057) checknative command to probe for winutils.exe on windows
[
https://issues.apache.org/jira/browse/HADOOP-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126444#comment-14126444
]
Hadoop QA commented on HADOOP-11057:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667289/HADOOP-11057.1.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4676//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4676//console
This message is automatically generated.
checknative command to probe for winutils.exe on windows
Key: HADOOP-11057
URL: https://issues.apache.org/jira/browse/HADOOP-11057
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 2.5.0
Environment: windows
Reporter: Steve Loughran
Assignee: Xiaoyu Yao
Priority: Minor
Fix For: 2.4.1
Attachments: HADOOP-11057.0.patch, HADOOP-11057.1.patch
hadoop's {{checknative}} command looks for native binaries and returns an
error code if one is missing.
But it doesn't check for {{winutils.exe}} on windows, which turns out to be
essential for some operations.
Adding this check to the -a (or default) operation would allow the check to
be used as a health check on windows installations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11077) NPE if hosts not specified in ProxyUsers
[
https://issues.apache.org/jira/browse/HADOOP-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126449#comment-14126449
]
Hadoop QA commented on HADOOP-11077:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667290/HADOOP-11077.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4677//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4677//console
This message is automatically generated.
NPE if hosts not specified in ProxyUsers
Key: HADOOP-11077
URL: https://issues.apache.org/jira/browse/HADOOP-11077
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.6.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Attachments: HADOOP-11077.patch
When using the TokenDelegationAuthenticationFilter, I noticed if I don't
specify the hosts for a user/groups proxy user and then try to authenticate,
I get an NPE rather than an AuthorizationException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[
https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11062:
-
Attachment: HADOOP-11062.2.patch
Uploading latest Patch.
[~cmccabe], [~clamb] , In the first patch, I had used Junit's
{{Assume.assumeTrue()}} function to which you can pass a boolean (in our case
the check if {{-Pnative}} is enabled) and a Message... and it should skip the
test after logging the message if the boolean is false .. Unfortunately, the
maven surefire plugin does not seem to be logging the message anywhere..
I have modified the patch to actually throw a {{LOG.warn()}}. The message will
be captured in the
{{target/surefire-reports/org.apache.hadoop.crypto-output.txt}} file
CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
--
Key: HADOOP-11062
URL: https://issues.apache.org/jira/browse/HADOOP-11062
Project: Hadoop Common
Issue Type: Bug
Components: security, test
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Arun Suresh
Attachments: HADOOP-11062.1.patch, HADOOP-11062.1.patch,
HADOOP-11062.2.patch
there are a few testcases, cryptocodec related that require Hadoop native
code and OpenSSL.
These tests should be skipped if -Pnative is not used when running the tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11044) FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps
[ https://issues.apache.org/jira/browse/HADOOP-11044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126465#comment-14126465 ] Gary Steelman commented on HADOOP-11044: Unfortunately I don't see immediately where the additional javac warnings are coming from. The way the Jira message is generated can be found in dev-support/test-patch.sh and the function checkJavacWarnings(). Can you try running that logic locally to see what the warnings are? FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps Key: HADOOP-11044 URL: https://issues.apache.org/jira/browse/HADOOP-11044 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.5.0, 2.4.1 Reporter: Swapnil Daingade Priority: Minor Attachments: 11044.patch4 The org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData class defines readOps, largeReadOps, writeOps as int. Also the The org.apache.hadoop.fs.FileSystem.Statistics class has methods like getReadOps(), getLargeReadOps() and getWriteOps() that return int. These int values can overflow if the exceed 2^31-1 showing negative values. It would be nice if these can be changed to long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[
https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126596#comment-14126596
]
Hadoop QA commented on HADOOP-11062:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667317/HADOOP-11062.2.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4678//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4678//console
This message is automatically generated.
CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
--
Key: HADOOP-11062
URL: https://issues.apache.org/jira/browse/HADOOP-11062
Project: Hadoop Common
Issue Type: Bug
Components: security, test
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Arun Suresh
Attachments: HADOOP-11062.1.patch, HADOOP-11062.1.patch,
HADOOP-11062.2.patch
there are a few testcases, cryptocodec related that require Hadoop native
code and OpenSSL.
These tests should be skipped if -Pnative is not used when running the tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
