[jira] [Resolved] (HADOOP-8392) Add YARN audit logging to log4j.properties
[ https://issues.apache.org/jira/browse/HADOOP-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved HADOOP-8392. - Resolution: Duplicate It sounds a YARN issue. There already exists a YARN jira: YARN-2255. So close this jira as duplicate of that one. Add YARN audit logging to log4j.properties -- Key: HADOOP-8392 URL: https://issues.apache.org/jira/browse/HADOOP-8392 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins MAPREDUCE-2655 added MR/NM audit logging but it's not hooked up into log4j.properties or the bin and env scripts like the other audit logs so you have to modify the deployed binary to change them. Let's add the relevant plumbing that the other audit loggers have, and update log4.properties with a sample configuration that's disabled by default, eg see [this comment|https://issues.apache.org/jira/browse/MAPREDUCE-2655?focusedCommentId=13084191page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13084191]. Also, looks like mapred.AuditLogger and its plumbing can be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11749) HttpServer2 thread pool is set to daemon
Zhijie Shen created HADOOP-11749: Summary: HttpServer2 thread pool is set to daemon Key: HADOOP-11749 URL: https://issues.apache.org/jira/browse/HADOOP-11749 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen In many cases, it is not the problem because the rpc protocol will bock the process from exit. However, if the process only has a web server, since the thread pool is set to daemon, the process will immediately exit after starting it, but not stay and listen to the incoming requests. It's possible for us to work around to make the main thread being blocked, but I'm wondering if we can resolve it within HttpServer2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11370) Fix new findbug warnings hadoop-yarn
[ https://issues.apache.org/jira/browse/HADOOP-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved HADOOP-11370. -- Resolution: Fixed Fix Version/s: 2.7.0 Assignee: (was: Varun Saxena) Close it as all incorporated Jiras are resolved. Unset the assignee because of multiple contributors. Fix new findbug warnings hadoop-yarn Key: HADOOP-11370 URL: https://issues.apache.org/jira/browse/HADOOP-11370 Project: Hadoop Common Issue Type: Sub-task Reporter: Zhijie Shen Fix For: 2.7.0 Attachments: FindBugs Report.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11370) Fix new findbug warnings hadoop-yarn
Zhijie Shen created HADOOP-11370: Summary: Fix new findbug warnings hadoop-yarn Key: HADOOP-11370 URL: https://issues.apache.org/jira/browse/HADOOP-11370 Project: Hadoop Common Issue Type: Sub-task Reporter: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11254) Promoting AccessControlList to be public
Zhijie Shen created HADOOP-11254: Summary: Promoting AccessControlList to be public Key: HADOOP-11254 URL: https://issues.apache.org/jira/browse/HADOOP-11254 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen The motivation of promoting AccessControlList to be a public API is to facilitate the users to programmatically parse or construct an ACL string. A typical use case may be the timeline domain, where we have the client lib to accept strings complied with ACL string format to define the authorized readers/writers. It will be more convenient and less buggy if users can compose this string with AccessControlList. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11240) Jenkins build seems to be broken by changes in test-patch.sh
Zhijie Shen created HADOOP-11240:
Summary: Jenkins build seems to be broken by changes in
test-patch.sh
Key: HADOOP-11240
URL: https://issues.apache.org/jira/browse/HADOOP-11240
Project: Hadoop Common
Issue Type: Bug
Reporter: Zhijie Shen
Priority: Blocker
* https://builds.apache.org/job/PreCommit-YARN-Build/5596//console
* https://builds.apache.org/job/PreCommit-YARN-Build/5595//console
* https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4981//console
A couple jenkins build failure for the same reason:
{code}
HEAD is now at b0e19c9 HADOOP-10926. Improve test-patch.sh to apply binary
diffs (cmccabe)
Previous HEAD position was b0e19c9... HADOOP-10926. Improve test-patch.sh to
apply binary diffs (cmccabe)
Switched to branch 'trunk'
Your branch is behind 'origin/trunk' by 17 commits, and can be fast-forwarded.
(use git pull to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded trunk to b0e19c9d54cecef191b91431f9ca62a76a000f45.
MAPREDUCE-5933 patch is being downloaded at Tue Oct 28 02:11:12 UTC 2014 from
http://issues.apache.org/jira/secure/attachment/12677496/MAPREDUCE-5933.patch
cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory
Error: Patch dryrun couldn't detect changes the patch would make. Exiting.
PATCH APPLICATION FAILED
{code}
It seems to have been broken by HADOOP-10926
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11215) DT management ops in DelegationTokenAuthenticatedURL assume the authenticator is KerberosDelegationTokenAuthenticator
Zhijie Shen created HADOOP-11215:
Summary: DT management ops in DelegationTokenAuthenticatedURL
assume the authenticator is KerberosDelegationTokenAuthenticator
Key: HADOOP-11215
URL: https://issues.apache.org/jira/browse/HADOOP-11215
Project: Hadoop Common
Issue Type: Bug
Reporter: Zhijie Shen
Here's the code in get/renew/cancel DT:
{code}
return ((KerberosDelegationTokenAuthenticator) getAuthenticator()).
renewDelegationToken(url, token, token.delegationToken, doAsUser);
{code}
It seems not to be right because PseudoDelegationTokenAuthenticator should work
here as well. At least, it is inconsistent in the context of delegation token
authentication, as DelegationTokenAuthenticationHandler doesn't require the
authentication must be Kerberos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11210) Findbugs warning about SpanReceiverHost
Zhijie Shen created HADOOP-11210:
Summary: Findbugs warning about SpanReceiverHost
Key: HADOOP-11210
URL: https://issues.apache.org/jira/browse/HADOOP-11210
Project: Hadoop Common
Issue Type: Bug
Reporter: Zhijie Shen
Priority: Minor
Findbugs warning about SpanReceiverHost has bee reported multiple times in some
Jira:
{quote}
Dereference of the result of readLine() without nullcheck in
org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName()
Bug type NP_DEREFERENCE_OF_READLINE_VALUE (click for details)
In class org.apache.hadoop.tracing.SpanReceiverHost
In method
org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName()
Value loaded from line
At SpanReceiverHost.java:[line 104]
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11207) DelegationTokenAuthenticationHandler needs to support DT operations for proxy user
Zhijie Shen created HADOOP-11207: Summary: DelegationTokenAuthenticationHandler needs to support DT operations for proxy user Key: HADOOP-11207 URL: https://issues.apache.org/jira/browse/HADOOP-11207 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Zhijie Shen Assignee: Zhijie Shen Currently, DelegationTokenAuthenticationHandler only support DT operations for the request user after it passes the authentication. However, it should also support the request user to do DT operations on behalf of the proxy user. Timeline server is using the authentication filter for DT operations instead of traditional RPC-based ones. It needs this feature to enable the proxy user to use the timeline service (YARN-2676). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11181) o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier
Zhijie Shen created HADOOP-11181:
Summary: o.a.h.security.token.delegation.DelegationTokenManager
should be more generalized to handle other DelegationTokenIdentifier
Key: HADOOP-11181
URL: https://issues.apache.org/jira/browse/HADOOP-11181
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
While DelegationTokenManager can set external secretManager, it have the
assumption that the token is going to be
o.a.h.security.token.delegation.DelegationTokenIdentifier, and use
DelegationTokenIdentifier method to decode a token.
{code}
@SuppressWarnings(unchecked)
public UserGroupInformation verifyToken(TokenDelegationTokenIdentifier
token) throws IOException {
ByteArrayInputStream buf = new ByteArrayInputStream(token.getIdentifier());
DataInputStream dis = new DataInputStream(buf);
DelegationTokenIdentifier id = new DelegationTokenIdentifier(tokenKind);
id.readFields(dis);
dis.close();
secretManager.verifyToken(id, token.getPassword());
return id.getUser();
}
{code}
It's not going to work it the token kind is other than
web.DelegationTokenIdentifier. For example, RM want to reuse it but hook it to
RMDelegationTokenSecretManager and RMDelegationTokenIdentifier, which has the
customized way to decode a token.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-10995) HBase cannot run correctly with Hadoop trunk
[ https://issues.apache.org/jira/browse/HADOOP-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved HADOOP-10995. -- Resolution: Invalid Release Note: Per the new patch on YARN-2032, we plan to ignore the test cases on trunk to walk around the compatibility issue. Close this ticket as invalid. HBase cannot run correctly with Hadoop trunk Key: HADOOP-10995 URL: https://issues.apache.org/jira/browse/HADOOP-10995 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Critical Attachments: HADOOP-10995.1.patch, YARN-2032.dependency.patch Several incompatible changes that happened on trunk but not on branch-2 have broken the compatibility for HBbase: HADOOP-10348 HADOOP-8124 HADOOP-10255 In general, HttpServer is and Syncable.sync have been missed. It blocks YARN-2032, which makes timeline sever support HBase store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-10995) HBase cannot run correctly with Hadoop trunk
Zhijie Shen created HADOOP-10995: Summary: HBase cannot run correctly with Hadoop trunk Key: HADOOP-10995 URL: https://issues.apache.org/jira/browse/HADOOP-10995 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Critical Several incompatible changes that happened on trunk but not on branch-2 have broken the compatibility for HBbase: HADOOP-10348 HADOOP-8124 HADOOP-10255 In general, HttpServer is and Syncable.sync have been missed. It blocks YARN-2032, which makes timeline sever support HBase store. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10794) A hadoop cluster needs clock synchronization
Zhijie Shen created HADOOP-10794: Summary: A hadoop cluster needs clock synchronization Key: HADOOP-10794 URL: https://issues.apache.org/jira/browse/HADOOP-10794 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen As a distributed system, a hadoop cluster wants the clock on all the participating hosts synchronized. Otherwise, some problems might happen. For example, in MAPREDUCE-5940, due to the clock on the host for the task container falls behind that on the host of the AM container, the computed elapsed time (the diff between the timestamps produced on two hosts) becomes negative. In MAPREDUCE-5940, we tried to mask the negative elapsed time. However, we should seek for a decent long term solution, such as providing mechanism to do and check clock synchronization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10718) IOException: An existing connection was forcibly closed by the remote host frequently happens on Windows
Zhijie Shen created HADOOP-10718:
Summary: IOException: An existing connection was forcibly closed
by the remote host frequently happens on Windows
Key: HADOOP-10718
URL: https://issues.apache.org/jira/browse/HADOOP-10718
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Reporter: Zhijie Shen
After HADOOP-317, we still observed that on windows platform, there're a number
of IOException: An existing connection was forcibly closed by the remote host
when running a MR job. For example,
{code}
2014-06-09 09:11:40,675 INFO [Socket Reader #3 for port 59622]
org.apache.hadoop.ipc.Server: Socket Reader #3 for port 59622: readAndProcess
from client 10.215.30.53 threw exception [java.io.IOException: An existing
connection was forcibly closed by the remote host]
java.io.IOException: An existing connection was forcibly closed by the remote
host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
at org.apache.hadoop.ipc.Server.channelRead(Server.java:2558)
at org.apache.hadoop.ipc.Server.access$2800(Server.java:130)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1459)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:750)
at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:624)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:595)
{code}
{code}
2014-06-09 09:15:38,539 WARN [main] org.apache.hadoop.mapred.Task: Failure
sending commit pending: java.io.IOException: Failed on local exception:
java.io.IOException: An existing connection was forcibly closed by the remote
host; Host Details : local host is: sdevin-clster53/10.215.16.72; destination
host is: sdevin-clster54:63415;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1414)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)
at com.sun.proxy.$Proxy9.commitPending(Unknown Source)
at org.apache.hadoop.mapred.Task.done(Task.java:1006)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:397)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: An existing connection was forcibly closed by
the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:510)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1054)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
{code}
And the latter one results in the issue of MAPREDUCE-5924.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10600) AuthenticationFilterInitializer doesn't allow null signature secret file
Zhijie Shen created HADOOP-10600: Summary: AuthenticationFilterInitializer doesn't allow null signature secret file Key: HADOOP-10600 URL: https://issues.apache.org/jira/browse/HADOOP-10600 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen AuthenticationFilterInitializer doesn't allow null signature secret file. However, null signature secret is acceptable in AuthenticationFilter, and a random signature secret is going to be created instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10596) HttpServer2 should apply the authentication filter to some urls instead of null
Zhijie Shen created HADOOP-10596: Summary: HttpServer2 should apply the authentication filter to some urls instead of null Key: HADOOP-10596 URL: https://issues.apache.org/jira/browse/HADOOP-10596 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen HttpServer2 should apply the authentication filter to some urls instead of null. In addition, it should be more flexible for users to configure SPNEGO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10601) We should prevent AuthenticationFilter to be installed twice
Zhijie Shen created HADOOP-10601: Summary: We should prevent AuthenticationFilter to be installed twice Key: HADOOP-10601 URL: https://issues.apache.org/jira/browse/HADOOP-10601 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen It seems that we have two way to install the authentication filter (at least from YARN aspect): 1. have SPNEGO configs and use withHttpSpnego when starting the web app; 2. add AuthenticationFilterInitializer into the configuration of filter initializer list. If both ways are used, it seems that two AuthenticationFilter will be instantiated, which is not expected. It's good to allow one or the other. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10550) HttpAuthentication.html is out of date
Zhijie Shen created HADOOP-10550:
Summary: HttpAuthentication.html is out of date
Key: HADOOP-10550
URL: https://issues.apache.org/jira/browse/HADOOP-10550
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.0, 3.0.0
Reporter: Zhijie Shen
Priority: Minor
It is still saying:
{code}
By default Hadoop HTTP web-consoles (JobTracker, NameNode, TaskTrackers and
DataNodes) allow access without any form of authentication.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10397) No 64-bit native lib in Hadoop releases
Zhijie Shen created HADOOP-10397: Summary: No 64-bit native lib in Hadoop releases Key: HADOOP-10397 URL: https://issues.apache.org/jira/browse/HADOOP-10397 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen Recently, I had a chance to talk to a Hadoop user, who complained there's no 64-bit native lib in Hadaoop releases, and it was user unfriendly to make them download all the dependenies to build 64-bit themselves. Hence I checked the recent two releases, 2.2 and 2.3, whose native lib are both ELF 32-bit LSB shared objects. I'm not aware of the reason why we don't release 64-bit, but I'd like to open the ticket to tackle this issue given we didn't before. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-9658) SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when native code is not loaded
Zhijie Shen created HADOOP-9658:
---
Summary: SnappyCodec#checkNativeCodeLoaded may unexpectedly fail
when native code is not loaded
Key: HADOOP-9658
URL: https://issues.apache.org/jira/browse/HADOOP-9658
Project: Hadoop Common
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
{code}
public static void checkNativeCodeLoaded() {
if (!NativeCodeLoader.buildSupportsSnappy()) {
throw new RuntimeException(native snappy library not available: +
this version of libhadoop was built without +
snappy support.);
}
if (!SnappyCompressor.isNativeCodeLoaded()) {
throw new RuntimeException(native snappy library not available: +
SnappyCompressor has not been loaded.);
}
if (!SnappyDecompressor.isNativeCodeLoaded()) {
throw new RuntimeException(native snappy library not available: +
SnappyDecompressor has not been loaded.);
}
}
{code}
buildSupportsSnappy is native method. If the native code is not loaded, the
method will be missing. Therefore, whether the native code is loaded or not,
the first runtime exception will not be thrown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9538) Make AMLauncher in RM Use NMClient
Zhijie Shen created HADOOP-9538: --- Summary: Make AMLauncher in RM Use NMClient Key: HADOOP-9538 URL: https://issues.apache.org/jira/browse/HADOOP-9538 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen YARN-422 adds NMClient. RM's AMLauncher is responsible for the interactions with an application's AM container. AMLauncher should also replace the raw ContainerManager proxy with NMClient. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
