[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Target Version/s: 2.5.0 (was: 2.4.0)
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
Akira AJISAKA created HADOOP-10557: -- Summary: FsShell -cp -p does not preserve extended ACLs Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Reporter: Akira AJISAKA This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[
https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986436#comment-13986436
]
Akira AJISAKA commented on HADOOP-10557:
I perfer to preserve extended ACLs by {{-p}} option rather than creating a new
option because {{cp -p}} preserves extended ACLs in unix.
> FsShell -cp -p does not preserve extended ACLs
> --
>
> Key: HADOOP-10557
> URL: https://issues.apache.org/jira/browse/HADOOP-10557
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira AJISAKA
>
> This issue tracks enhancing FsShell cp to
> * preserve extended ACLs by -p option
> or
> * add a new command-line option for preserving extended ACLs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10552) Fix usage and example at FileSystemShell.apt.vm
[
https://issues.apache.org/jira/browse/HADOOP-10552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986438#comment-13986438
]
Kenji Kikushima commented on HADOOP-10552:
--
Thanks for comment, [~qwertymaniac].
My purpose is equalize style of usage and example, not to replacing hadoop fs.
In FileSystemShell.apt.vm, all other commands are described using hdfs dfs.
In addtion, I have a basic question about "hadoop fs" and "hdfs dfs". In
CommandsManual.apt.vm, hadoop fs section says
{noformat}
Usage: <<>>
Deprecated, use <<>> instead.
{noformat}
Is this incorrect? Should we revise "Deprecated, use <<>> instead."
to "You can also use <> equally."?
> Fix usage and example at FileSystemShell.apt.vm
> ---
>
> Key: HADOOP-10552
> URL: https://issues.apache.org/jira/browse/HADOOP-10552
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Kenji Kikushima
>Priority: Trivial
> Attachments: HADOOP-10552.patch
>
>
> Usage at moveFromLocal needs "hdfs" command, and example for touchz should
> use "hdfs dfs".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10492) Help Commands needs change after deprecation
[ https://issues.apache.org/jira/browse/HADOOP-10492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenji Kikushima updated HADOOP-10492: - Attachment: HADOOP-10492.patch Hi, I checked hadoop and hdfs commands. We have three ways to call FsShell, "hadoop fs" "hadoop dfs" and "hdfs dfs". Only "hadoop dfs" is deprecated. So, I think we should print "hadoop fs" and "hdfs dfs" like this patch. Thanks. > Help Commands needs change after deprecation > > > Key: HADOOP-10492 > URL: https://issues.apache.org/jira/browse/HADOOP-10492 > Project: Hadoop Common > Issue Type: Bug >Reporter: Raja Nagendra Kumar > Attachments: HADOOP-10492.patch > > > As hadoop dfs is deprecated, the help should show usage with HDFS > e.g in the following command it still refers to > Usage: hadoop fs [generic options] > D:\Apps\java\BI\hadoop\hw\hdp\hadoop-2.2.0.2.0.6.0-0009>hdfs dfs > Usage: hadoop fs [generic options] > [-appendToFile ... ] > [-cat [-ignoreCrc] ...] > [-checksum ...] > [-chgrp [-R] GROUP PATH...] > [-chmod [-R] PATH...] > [-chown [-R] [OWNER][:[GROUP]] PATH...] > [-copyFromLocal [-f] [-p] ... ] > [-copyToLocal [-p] [-ignoreCrc] [-crc] ... ] > [-count [-q] ...] > [-cp [-f] [-p] ... ] > [-createSnapshot []] > [-deleteSnapshot ] > [-df [-h] [ ...]] > [-du [-s] [-h] ...] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10556) Add toLowerCase support to auth_to_local rules for service name
[ https://issues.apache.org/jira/browse/HADOOP-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986462#comment-13986462 ] Steve Loughran commented on HADOOP-10556: - this'll be done in an i18n manner, presumably? > Add toLowerCase support to auth_to_local rules for service name > --- > > Key: HADOOP-10556 > URL: https://issues.apache.org/jira/browse/HADOOP-10556 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.4.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > > When using Vintela to integrate Linux with AD, principals are lowercased. If > the accounts in AD have uppercase characters (ie FooBar) the Kerberos > principals have also uppercase characters (ie FooBar/). Because of > this, when a service (Yarn/HDFS) extracts the service name from the Kerberos > principal (FooBar) and uses it for obtain groups the user is not found > because via Linux the user FooBar is unknown, it has been converted to foobar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
Sami Abobala created HADOOP-10558: - Summary: java.net.UnknownHostException: Invalid host name: local host is: (unknown) Key: HADOOP-10558 URL: https://issues.apache.org/jira/browse/HADOOP-10558 Project: Hadoop Common Issue Type: Bug Environment: two node cluster ubuntu 12.04 LTS in both Java version "1.7.0_25" node 1: core i5 4 GB RAM node 2: core i3 4 GB RAM Reporter: Sami Abobala I had this exception every time i try to run map-red job, I went to http://wiki.apache.org/hadoop/UnknownHost and tried every possible solution and still have the same result Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED Container launch failed for container_1398945803120_0001_01_06 : java.lang.reflect.UndeclaredThrowableException . . Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":8042; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost . . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9882) Trunk doesn't compile
[ https://issues.apache.org/jira/browse/HADOOP-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986552#comment-13986552 ] Ritesh Kalra commented on HADOOP-9882: -- Hi Jean you already must have protoc-2.4.1 installed. (protoc --version). Try upgrading to 2.5.0 1. Download protocol buffer. 2. Check if g++ compiler is installed on box. Protocol buffer needs g++ compiler to be present on your box before it can be builded. This is a crisp post on how to install g++ compiler on your box. Install g++ compiler. 3. Extract the protocol buffer archive and switch to the extracted directory. 4. Inside the extracted directory hit the below commands to install protocol buffer. These may take a while, kindly be patient. $ sudo ./configure $ sudo make $ sudo make check $ sudo make install $ protoc --version > Trunk doesn't compile > - > > Key: HADOOP-9882 > URL: https://issues.apache.org/jira/browse/HADOOP-9882 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Jean-Baptiste Onofré > > Currently, trunk does not compile (in hadoop-common-project/hadoop-common > module): > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > protoc version is 'libprotoc 2.4.1', expected version is '2.5.0' -> [Help 1] > I gonna fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Abobala updated HADOOP-10558: -- Labels: hadoop mapreduce (was: ) > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: hadoop, mapreduce > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Abobala updated HADOOP-10558: -- Labels: hadoop mapreduce ubuntu (was: hadoop mapreduce) > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: hadoop, mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10307) Support multiple Authentication mechanisms for HTTP
[ https://issues.apache.org/jira/browse/HADOOP-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986649#comment-13986649 ] Kihwal Lee commented on HADOOP-10307: - It will be nice if it allows to try auth filters in the chain one by one until one succeeds. The default one should be the first one. If the client specified one or more preferred methods, only those would be tried. I haven't looked at the code, but we may need a container auth filter to support trying multiple auth methods. > Support multiple Authentication mechanisms for HTTP > --- > > Key: HADOOP-10307 > URL: https://issues.apache.org/jira/browse/HADOOP-10307 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10307.patch > > > Currently it is possible to specify a custom Authentication Handler for HTTP > authentication. > We have a requirement to support multiple mechanisms to authenticate HTTP > access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HADOOP-10516) the NodeManager in slave does not starting
[ https://issues.apache.org/jira/browse/HADOOP-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Abobala reopened HADOOP-10516: --- I edit the cluster ID in DataNode and NameNode and make them the same. > the NodeManager in slave does not starting > -- > > Key: HADOOP-10516 > URL: https://issues.apache.org/jira/browse/HADOOP-10516 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.23.9 > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala >Priority: Critical > Labels: cloud, hadoop, yarn > > the Node Manager cannot start in the slave node and this is the log: > 2014-04-17 14:47:54,217 FATAL > org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting > NodeManager > org.apache.hadoop.yarn.YarnException: Failed to Start > org.apache.hadoop.yarn.server.nodemanager.NodeManager > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:205) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:328) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:370) > Caused by: org.apache.avro.AvroRuntimeException: > java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:139) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > ... 3 more > Caused by: java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66) > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:176) > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:135) > ... 4 more > Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: > Call From fatima-HP-ProBook-4520s/127.0.1.1 to 0.0.0.0:8031 failed $ > at > org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144) > at com.sun.proxy.$Proxy24.registerNodeManager(Unknown Source) > at > org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59) > ... 6 more > Caused by: java.net.ConnectException: Call From > fatima-HP-ProBook-4520s/127.0.1.1 to 0.0.0.0:8031 failed on connection > exception: java.net.Conn$ > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:700) > at org.apache.hadoop.ipc.Client.call(Client.java:1098) > at > org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141) > ... 8 more > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) > at > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:462) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:559) > at > org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:207) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1204) > at org.apache.hadoop.ipc.Client.call(Client.java:1074) > ... 9 more > 2014-04-17 14:47:54,221 INFO org.apache.hadoop.ipc.Server: Stopping server on > 41790 > 2014-04-17 14:47:54,222 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager > metrics system.. > 2014-04-17 14:47:54,222 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager > metrics system... > 2014-04-17 14:47:54,224 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system > stopped. > 2014-04-17 14:47:54,224 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system > shutdown complete. > 2014-04-17 14:47:54,224 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: > / -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10516) the NodeManager in slave does not starting
[ https://issues.apache.org/jira/browse/HADOOP-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Abobala resolved HADOOP-10516. --- Resolution: Fixed > the NodeManager in slave does not starting > -- > > Key: HADOOP-10516 > URL: https://issues.apache.org/jira/browse/HADOOP-10516 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.23.9 > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala >Priority: Critical > Labels: cloud, hadoop, yarn > > the Node Manager cannot start in the slave node and this is the log: > 2014-04-17 14:47:54,217 FATAL > org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting > NodeManager > org.apache.hadoop.yarn.YarnException: Failed to Start > org.apache.hadoop.yarn.server.nodemanager.NodeManager > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:205) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:328) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:370) > Caused by: org.apache.avro.AvroRuntimeException: > java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:139) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > ... 3 more > Caused by: java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66) > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:176) > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:135) > ... 4 more > Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: > Call From fatima-HP-ProBook-4520s/127.0.1.1 to 0.0.0.0:8031 failed $ > at > org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144) > at com.sun.proxy.$Proxy24.registerNodeManager(Unknown Source) > at > org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59) > ... 6 more > Caused by: java.net.ConnectException: Call From > fatima-HP-ProBook-4520s/127.0.1.1 to 0.0.0.0:8031 failed on connection > exception: java.net.Conn$ > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:700) > at org.apache.hadoop.ipc.Client.call(Client.java:1098) > at > org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141) > ... 8 more > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) > at > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:462) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:559) > at > org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:207) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1204) > at org.apache.hadoop.ipc.Client.call(Client.java:1074) > ... 9 more > 2014-04-17 14:47:54,221 INFO org.apache.hadoop.ipc.Server: Stopping server on > 41790 > 2014-04-17 14:47:54,222 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager > metrics system.. > 2014-04-17 14:47:54,222 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager > metrics system... > 2014-04-17 14:47:54,224 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system > stopped. > 2014-04-17 14:47:54,224 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system > shutdown complete. > 2014-04-17 14:47:54,224 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: > / -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10158) SPNEGO should work with multiple interfaces/SPNs.
[ https://issues.apache.org/jira/browse/HADOOP-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986722#comment-13986722 ] Daryn Sharp commented on HADOOP-10158: -- Native code loader test failure is unrelated. > SPNEGO should work with multiple interfaces/SPNs. > - > > Key: HADOOP-10158 > URL: https://issues.apache.org/jira/browse/HADOOP-10158 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Kihwal Lee >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-10158-readkeytab.patch, > HADOOP-10158-readkeytab.patch, HADOOP-10158.patch, HADOOP-10158.patch, > HADOOP-10158.patch, HADOOP-10158_multiplerealms.patch, > HADOOP-10158_multiplerealms.patch, HADOOP-10158_multiplerealms.patch > > > This is the list of internal servlets added by namenode. > | Name | Auth | Need to be accessible by end users | > | StartupProgressServlet | none | no | > | GetDelegationTokenServlet | internal SPNEGO | yes | > | RenewDelegationTokenServlet | internal SPNEGO | yes | > | CancelDelegationTokenServlet | internal SPNEGO | yes | > | FsckServlet | internal SPNEGO | yes | > | GetImageServlet | internal SPNEGO | no | > | ListPathsServlet | token in query | yes | > | FileDataServlet | token in query | yes | > | FileChecksumServlets | token in query | yes | > | ContentSummaryServlet | token in query | yes | > GetDelegationTokenServlet, RenewDelegationTokenServlet, > CancelDelegationTokenServlet and FsckServlet are accessed by end users, but > hard-coded to use the internal SPNEGO filter. > If a name node HTTP server binds to multiple external IP addresses, the > internal SPNEGO service principal name may not work with an address to which > end users are connecting. The current SPNEGO implementation in Hadoop is > limited to use a single service principal per filter. > If the underlying hadoop kerberos authentication handler cannot easily be > modified, we can at least create a separate auth filter for the end-user > facing servlets so that their service principals can be independently > configured. If not defined, it should fall back to the current behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10559) add a method to UserGroupInformation to load settings from a given conf file
Steve Loughran created HADOOP-10559:
---
Summary: add a method to UserGroupInformation to load settings
from a given conf file
Key: HADOOP-10559
URL: https://issues.apache.org/jira/browse/HADOOP-10559
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 2.4.0
Reporter: Steve Loughran
There's no easy way to set up the security parameters of a process unless it's
set in core-site.xml, because it's just inited via:
{{initialize(new Configuration(), false);}}
# If it is defined in an XML resource injected in to the config resource list
via {{Configuration.addResource()}} -then it *may* get picked up, but only if
nothing has already created the configs.
# If it is defined in any other means -you can't get it in.
This is an issue with client apps that don't have core-site XML files on their
classpath, and which are loading their configs more dynamically. Everything
works on an insecure cluster, but try to target a secure one and things break.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10559) add a method to UserGroupInformation to load settings from a given conf file
[
https://issues.apache.org/jira/browse/HADOOP-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986745#comment-13986745
]
Steve Loughran commented on HADOOP-10559:
-
There's an {{@VisibleForTesting}} reset() method, but that still takes the
shared config. I'd rather some {{reload(Configuration)}} method that would
reset everything and reload the details from the given config.
> add a method to UserGroupInformation to load settings from a given conf file
>
>
> Key: HADOOP-10559
> URL: https://issues.apache.org/jira/browse/HADOOP-10559
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>
> There's no easy way to set up the security parameters of a process unless
> it's set in core-site.xml, because it's just inited via:
> {{initialize(new Configuration(), false);}}
> # If it is defined in an XML resource injected in to the config resource list
> via {{Configuration.addResource()}} -then it *may* get picked up, but only if
> nothing has already created the configs.
> # If it is defined in any other means -you can't get it in.
> This is an issue with client apps that don't have core-site XML files on
> their classpath, and which are loading their configs more dynamically.
> Everything works on an insecure cluster, but try to target a secure one and
> things break.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10559) add a method to UserGroupInformation to load settings from a given conf file
[
https://issues.apache.org/jira/browse/HADOOP-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986747#comment-13986747
]
Steve Loughran commented on HADOOP-10559:
-
correction {{addDefaultResource()}} is the way to force in a new conf file on
the classpath -but only before the first security init, and only if the
settings are on the classpath
> add a method to UserGroupInformation to load settings from a given conf file
>
>
> Key: HADOOP-10559
> URL: https://issues.apache.org/jira/browse/HADOOP-10559
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>
> There's no easy way to set up the security parameters of a process unless
> it's set in core-site.xml, because it's just inited via:
> {{initialize(new Configuration(), false);}}
> # If it is defined in an XML resource injected in to the config resource list
> via {{Configuration.addResource()}} -then it *may* get picked up, but only if
> nothing has already created the configs.
> # If it is defined in any other means -you can't get it in.
> This is an issue with client apps that don't have core-site XML files on
> their classpath, and which are loading their configs more dynamically.
> Everything works on an insecure cluster, but try to target a secure one and
> things break.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10559) add a method to UserGroupInformation to load settings from a given conf file
[
https://issues.apache.org/jira/browse/HADOOP-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986754#comment-13986754
]
Steve Loughran commented on HADOOP-10559:
-
Looking at the code, I'm not being clear about this. The core issue is that the
UGI returned in the {{getLoginUser()}} can can only be initialized once -and if
it is done before the client gets to push down their desired configuration,
with their desired security options, you end up with simple auth that doesn't
work with a secure cluster
> add a method to UserGroupInformation to load settings from a given conf file
>
>
> Key: HADOOP-10559
> URL: https://issues.apache.org/jira/browse/HADOOP-10559
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>
> There's no easy way to set up the security parameters of a process unless
> it's set in core-site.xml, because it's just inited via:
> {{initialize(new Configuration(), false);}}
> # If it is defined in an XML resource injected in to the config resource list
> via {{Configuration.addResource()}} -then it *may* get picked up, but only if
> nothing has already created the configs.
> # If it is defined in any other means -you can't get it in.
> This is an issue with client apps that don't have core-site XML files on
> their classpath, and which are loading their configs more dynamically.
> Everything works on an insecure cluster, but try to target a secure one and
> things break.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
Ted Malaska created HADOOP-10560:
Summary: Update NativeS3FileSystem to issue copy commands for
files with in a directory with a configurable number of threads
Key: HADOOP-10560
URL: https://issues.apache.org/jira/browse/HADOOP-10560
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Reporter: Ted Malaska
Priority: Minor
In NativeS3FileSystem if you do a copy of a directory it will copy all the
files to the new location, but it will do this with one thread. Code is below.
This jira will allow a configurable number of threads to be used to issue the
copy commands to S3.
do {
PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
priorLastKey, true);
for (FileMetadata file : listing.getFiles())
{ keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
file.getKey().substring(srcKey.length())); }
priorLastKey = listing.getPriorLastKey();
} while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986824#comment-13986824
]
Ted Malaska commented on HADOOP-10560:
--
If you don't mind I would like to do this Jira. I'm being set up as a
contributor now. I will assign it to myself as soon as that is finished.
Thanks
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
>
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Ted Malaska
>Priority: Minor
> Labels: performance
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10561) Copy command with preserve option should handle Xattrs
Uma Maheswara Rao G created HADOOP-10561: Summary: Copy command with preserve option should handle Xattrs Key: HADOOP-10561 URL: https://issues.apache.org/jira/browse/HADOOP-10561 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Uma Maheswara Rao G The design docs for Xattrs stated that we handle preserve options with copy commands >From doc: Preserve option of commands like “cp -p” shell command and “distcp -p” should work on XAttrs. In the case of source fs supports XAttrs but target fs does not support, XAttrs will be ignored with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[
https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986873#comment-13986873
]
Uma Maheswara Rao G commented on HADOOP-10561:
--
Current code of copy command preserve options handled for owner. permission.
times:
{code}
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src
{code}
> Copy command with preserve option should handle Xattrs
> --
>
> Key: HADOOP-10561
> URL: https://issues.apache.org/jira/browse/HADOOP-10561
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: Uma Maheswara Rao G
>
> The design docs for Xattrs stated that we handle preserve options with copy
> commands
> From doc:
> Preserve option of commands like “cp -p” shell command and “distcp -p” should
> work on XAttrs.
> In the case of source fs supports XAttrs but target fs does not support,
> XAttrs will be ignored
> with warning message
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HADOOP-10561: - Assignee: Yi Liu > Copy command with preserve option should handle Xattrs > -- > > Key: HADOOP-10561 > URL: https://issues.apache.org/jira/browse/HADOOP-10561 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > > The design docs for Xattrs stated that we handle preserve options with copy > commands > From doc: > Preserve option of commands like “cp -p” shell command and “distcp -p” should > work on XAttrs. > In the case of source fs supports XAttrs but target fs does not support, > XAttrs will be ignored > with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza updated HADOOP-10560:
Assignee: Ted Malaska
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
>
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Labels: performance
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986936#comment-13986936
]
Owen O'Malley commented on HADOOP-10433:
Alejandro,
A couple more things on the interface:
* You need to allow the user to specify the key-name when they create the key.
* Instead of throwing if the url is too long, please break the user's request
into parts that fit within the 2000 byte limit.
* You never answered whether the body is xml or json (or both).
* The underscores should be on the trailing part of the url.
I'd propose it look like:
{code}
create key : PUThttp://HOST:PORT/kms/v1/key/
rollover key : POST http://HOST:PORT/kms/v1/key/
delete key : DELETE http://HOST:PORT/kms/v1/key/
get metadata : GEThttp://HOST:PORT/kms/v1/key//_metadata
get current version: GET
http://HOST:PORT/kms/v1/key//_currentversion
get key versions : GEThttp://HOST:PORT/kms/v1/key//_versions
get key version: GEThttp://HOST:PORT/kms/v1/keyversion/
get key names : GEThttp://HOST:PORT/kms/v1/keys/names
get keys metadata : GET
http://HOST:PORT/kms/v1/keys/metadata?key=&key=,...
{code}
> Key Management Server based on KeyProvider API
> --
>
> Key: HADOOP-10433
> URL: https://issues.apache.org/jira/browse/HADOOP-10433
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HadoopKMSDocsv2.pdf, KMS-doc.pdf
>
>
> (from HDFS-6134 proposal)
> Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying
> KMS. It provides an interface that works with existing Hadoop security
> components (authenticatication, confidentiality).
> Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141
> and HADOOP-10177.
> Hadoop KMS will provide an additional implementation of the Hadoop
> KeyProvider class. This implementation will be a client-server implementation.
> The client-server protocol will be secure:
> * Kerberos HTTP SPNEGO (authentication)
> * HTTPS for transport (confidentiality and integrity)
> * Hadoop ACLs (authorization)
> The Hadoop KMS implementation will not provide additional ACL to access
> encrypted files. For sophisticated access control requirements, HDFS ACLs
> (HDFS-4685) should be used.
> Basic key administration will be supported by the Hadoop KMS via the, already
> available, Hadoop KeyShell command line tool
> There are minor changes that must be done in Hadoop KeyProvider functionality:
> The KeyProvider contract, and the existing implementations, must be
> thread-safe
> KeyProvider API should have an API to generate the key material internally
> JavaKeyStoreProvider should use, if present, a password provided via
> configuration
> KeyProvider Option and Metadata should include a label (for easier
> cross-referencing)
> To avoid overloading the underlying KeyProvider implementation, the Hadoop
> KMS will cache keys using a TTL policy.
> Scalability and High Availability of the Hadoop KMS can achieved by running
> multiple instances behind a VIP/Load-Balancer. For High Availability, the
> underlying KeyProvider implementation used by the Hadoop KMS must be High
> Available.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Moved] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas moved HDFS-6320 to HADOOP-10562: Component/s: (was: namenode) Affects Version/s: (was: 1.2.1) (was: 2.0.0-alpha) 2.0.0-alpha 1.2.1 Key: HADOOP-10562 (was: HDFS-6320) Project: Hadoop Common (was: Hadoop HDFS) > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0-alpha >Reporter: Suresh Srinivas > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-10562: - Priority: Critical (was: Major) > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Priority: Critical > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-10562: - Attachment: HADOOP-10562.patch Patch to print the exception stack trace > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-10562: - Status: Patch Available (was: Open) > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0-alpha >Reporter: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10279) Create multiplexer, a requirement for the fair queue
[ https://issues.apache.org/jira/browse/HADOOP-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10279: -- Attachment: HADOOP-10279.patch Upload latest version of patch for reference > Create multiplexer, a requirement for the fair queue > > > Key: HADOOP-10279 > URL: https://issues.apache.org/jira/browse/HADOOP-10279 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li >Assignee: Chris Li > Attachments: HADOOP-10279.patch, WeightedRoundRobinMultiplexer.java, > subtask2_add_mux.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-10562: --- Assignee: Suresh Srinivas > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10562: --- Attachment: HADOOP-10562.branch-1.patch +1 for the patch. Also attached branch-1 patch. > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.branch-1.patch, HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Malaska updated HADOOP-10560:
-
Attachment: HADOOP-10560.patch
First attempt to added threading to copying files in the case a directory is
renamed.
I just use and executorService with futures. Then try to get all the futures
before preforming the normal deletes that were there before.
This is ready for review.
Thanks again
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
>
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Labels: performance
> Attachments: HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986993#comment-13986993 ] Arpit Agarwal commented on HADOOP-10562: I will commit this to trunk and branch-2 shortly. > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.branch-1.patch, HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10563) Remove the dependency of jsp in trunk
Haohui Mai created HADOOP-10563: --- Summary: Remove the dependency of jsp in trunk Key: HADOOP-10563 URL: https://issues.apache.org/jira/browse/HADOOP-10563 Project: Hadoop Common Issue Type: Improvement Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HADOOP-10563.000.patch After HDFS-6252 neither hdfs nor yarn uses jsp, thus the dependency of the jsp can be removed from the pom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10563) Remove the dependency of jsp in trunk
[ https://issues.apache.org/jira/browse/HADOOP-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HADOOP-10563: Attachment: HADOOP-10563.000.patch > Remove the dependency of jsp in trunk > - > > Key: HADOOP-10563 > URL: https://issues.apache.org/jira/browse/HADOOP-10563 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HADOOP-10563.000.patch > > > After HDFS-6252 neither hdfs nor yarn uses jsp, thus the dependency of the > jsp can be removed from the pom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10563) Remove the dependency of jsp in trunk
[ https://issues.apache.org/jira/browse/HADOOP-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HADOOP-10563: Status: Patch Available (was: Open) > Remove the dependency of jsp in trunk > - > > Key: HADOOP-10563 > URL: https://issues.apache.org/jira/browse/HADOOP-10563 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HADOOP-10563.000.patch > > > After HDFS-6252 neither hdfs nor yarn uses jsp, thus the dependency of the > jsp can be removed from the pom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Malaska updated HADOOP-10560:
-
Release Note: HADOOP-10560.patch is ready for review.
Status: Patch Available (was: Open)
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
>
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Labels: performance
> Attachments: HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10555) add offset support to MurmurHash
[ https://issues.apache.org/jira/browse/HADOOP-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-10555: --- Assignee: Sergey Shelukhin > add offset support to MurmurHash > > > Key: HADOOP-10555 > URL: https://issues.apache.org/jira/browse/HADOOP-10555 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Trivial > Attachments: HADOOP-10555.patch > > > From HIVE-6430 code review -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10564) Add username to native RPCv9 client
Colin Patrick McCabe created HADOOP-10564: - Summary: Add username to native RPCv9 client Key: HADOOP-10564 URL: https://issues.apache.org/jira/browse/HADOOP-10564 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Add the ability for the native RPCv9 client to set a username when initiating a connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-10562: - Attachment: HADOOP-10562.1.patch Slightly updated patch with two additions: # Print the current number of tokens (we saw namenode going out of memory while creating an array in this part of the code, this will help debug it) # Some code cleanup > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.1.patch, HADOOP-10562.branch-1.patch, > HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10564) Add username to native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-10564:
--
Attachment: HADOOP-10564.001.patch
This patch adds the ability to set the username in the IpcConnectionContext.
It does not yet implement authentication (SASL, etc.), but just setting that
field.
I also changed {{hrpc_proxy}} so that instances can be created on the stack.
This is helpful in the forthcoming RPC client, since it means that blocking
RPCs can just create one of these on the stack and make a call without having
to call {{malloc()}}.
> Add username to native RPCv9 client
> ---
>
> Key: HADOOP-10564
> URL: https://issues.apache.org/jira/browse/HADOOP-10564
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: HADOOP-10388
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10564.001.patch
>
>
> Add the ability for the native RPCv9 client to set a username when initiating
> a connection.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987015#comment-13987015 ] Owen O'Malley commented on HADOOP-10150: I've been working through this. We have two metadata items that we need for each file: * the key name and version * the iv Note that the current patches only store the iv, but we really need to store the key name and version. The version is absolutely critical because if you roll a new key version you don't want to re-write all of the current data. It seems to me there are three reasonable places to store the small amount of metadata: * at the beginning of the file * in a side file * encoded using a filename mangling scheme The beginning of the file creates trouble because it throws off the block calculations that are done by mapreduce. (In other words, if we slide all of the data down by 1k, then each input split will always cross HDFS block boundaries.) On the other hand, it doesn't add any load to the namenode and will always be consistent with the file. A side file doesn't change the offsets into the file, but does double the amount of traffic and storage required on the namenode. Doing name mangling means the underlying HDFS file names are more complicated, but it doesn't mess with either the file offsets or increase the load on the namenode. I think we should do the name mangling. What do others think? > Hadoop cryptographic file system > > > Key: HADOOP-10150 > URL: https://issues.apache.org/jira/browse/HADOOP-10150 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file > system-V2.docx, HADOOP cryptographic file system.pdf, > HDFSDataAtRestEncryptionAlternatives.pdf, > HDFSDataatRestEncryptionAttackVectors.pdf, > HDFSDataatRestEncryptionProposal.pdf, cfs.patch, extended information based > on INode feature.patch > > > There is an increasing need for securing data when Hadoop customers use > various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so > on. > HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based > on HADOOP “FilterFileSystem” decorating DFS or other file systems, and > transparent to upper layer applications. It’s configurable, scalable and fast. > High level requirements: > 1.Transparent to and no modification required for upper layer > applications. > 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if > the wrapped file system supports them. > 3.Very high performance for encryption and decryption, they will not > become bottleneck. > 4.Can decorate HDFS and all other file systems in Hadoop, and will not > modify existing structure of file system, such as namenode and datanode > structure if the wrapped file system is HDFS. > 5.Admin can configure encryption policies, such as which directory will > be encrypted. > 6.A robust key management framework. > 7.Support Pread and append operations if the wrapped file system supports > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10447) Implement C code for parsing Hadoop / HDFS URIs
[
https://issues.apache.org/jira/browse/HADOOP-10447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987017#comment-13987017
]
Chris Douglas commented on HADOOP-10447:
Will the native client will support loading other filesystem implementations by
scheme, like the familiar {{FileSystem}} and {{FileContext}} APIs?
> Implement C code for parsing Hadoop / HDFS URIs
> ---
>
> Key: HADOOP-10447
> URL: https://issues.apache.org/jira/browse/HADOOP-10447
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Colin Patrick McCabe
>
> We need some glue code for parsing Hadoop or HDFS URIs in C. Probably we
> should just put a small 'Path' wrapper around a URI parsing library like
> http://uriparser.sourceforge.net/ (BSD licensed)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HADOOP-10564) Add username to native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-10564 started by Colin Patrick McCabe. > Add username to native RPCv9 client > --- > > Key: HADOOP-10564 > URL: https://issues.apache.org/jira/browse/HADOOP-10564 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10564.001.patch > > > Add the ability for the native RPCv9 client to set a username when initiating > a connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987025#comment-13987025 ] Andrew Wang commented on HADOOP-10150: -- Hey Owen, I think the plan here is to use xattrs to store this additional data. Is that satisfactory? This means it wouldn't be a pure wrapper since it'd require the underlying filesystem to implement xattrs (HDFS-2006 is linked as "requires"). The upside is that the design is nicer, and we can do tighter integration with HDFS. > Hadoop cryptographic file system > > > Key: HADOOP-10150 > URL: https://issues.apache.org/jira/browse/HADOOP-10150 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file > system-V2.docx, HADOOP cryptographic file system.pdf, > HDFSDataAtRestEncryptionAlternatives.pdf, > HDFSDataatRestEncryptionAttackVectors.pdf, > HDFSDataatRestEncryptionProposal.pdf, cfs.patch, extended information based > on INode feature.patch > > > There is an increasing need for securing data when Hadoop customers use > various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so > on. > HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based > on HADOOP “FilterFileSystem” decorating DFS or other file systems, and > transparent to upper layer applications. It’s configurable, scalable and fast. > High level requirements: > 1.Transparent to and no modification required for upper layer > applications. > 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if > the wrapped file system supports them. > 3.Very high performance for encryption and decryption, they will not > become bottleneck. > 4.Can decorate HDFS and all other file systems in Hadoop, and will not > modify existing structure of file system, such as namenode and datanode > structure if the wrapped file system is HDFS. > 5.Admin can configure encryption policies, such as which directory will > be encrypted. > 6.A robust key management framework. > 7.Support Pread and append operations if the wrapped file system supports > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10281) Create a scheduler, which assigns schedulables a priority level
[ https://issues.apache.org/jira/browse/HADOOP-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10281: -- Attachment: HADOOP-10281.patch Upload latest version for reference > Create a scheduler, which assigns schedulables a priority level > --- > > Key: HADOOP-10281 > URL: https://issues.apache.org/jira/browse/HADOOP-10281 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li > Attachments: HADOOP-10281.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10562: --- Attachment: (was: HADOOP-10562.branch-1.patch) > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.1.patch, HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10282) Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls
[ https://issues.apache.org/jira/browse/HADOOP-10282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10282: -- Attachment: HADOOP-10282.patch Update with latest version of the FCQ to get preliminary feedback > Create a FairCallQueue: a multi-level call queue which schedules incoming > calls and multiplexes outgoing calls > -- > > Key: HADOOP-10282 > URL: https://issues.apache.org/jira/browse/HADOOP-10282 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li > Attachments: HADOOP-10282.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10562: --- Attachment: HADOOP-10562.branch-1.1.patch Patch for branch-1. > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.1.patch, HADOOP-10562.branch-1.1.patch, > HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10553) Performance: AM scaleability is 10% slower in 2.4 compared to 0.23.9
[ https://issues.apache.org/jira/browse/HADOOP-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987125#comment-13987125 ] Vinod Kumar Vavilapalli commented on HADOOP-10553: -- Note sure whether it is a MR issue or YARN issue yet. Moving to YARN for now. > Performance: AM scaleability is 10% slower in 2.4 compared to 0.23.9 > > > Key: HADOOP-10553 > URL: https://issues.apache.org/jira/browse/HADOOP-10553 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: patrick white > > Performance comparison benchmarks from 2.x against 0.23 shows AM scalability > benchmark's runtime is approximately 10% slower in 2.4.0. The trend is > consistent across later releases in both lines, latest release numbers are: > 2.4.0.0 runtime 255.6 seconds (avg 5 passes) > 0.23.9.12 runtime 230.4 seconds (avg 5 passes) > Diff: -9.9% > AM Scalability test is essentially a sleep job that measures time to launch > and complete a large number of mappers. > The diff is consistent and has been reproduced in both a larger (350 node, > 100,000 mappers) perf environment, as well as a small (10 node, 2,900 > mappers) demo cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987166#comment-13987166
]
Alejandro Abdelnur commented on HADOOP-10433:
-
Owen,
bq. You need to allow the user to specify the key-name when they create the key.
It is happening, it goes in the JSON payload of the create operation. Look in
the {{createKeyInternal()}} method
bq. You never answered whether the body is xml or json (or both).
Request/response bodies are JSON.
I'll integrate your other 2 suggestions, the underscore and the break into
multiple requests if URL goes beyond 2K byte limit.
> Key Management Server based on KeyProvider API
> --
>
> Key: HADOOP-10433
> URL: https://issues.apache.org/jira/browse/HADOOP-10433
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HadoopKMSDocsv2.pdf, KMS-doc.pdf
>
>
> (from HDFS-6134 proposal)
> Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying
> KMS. It provides an interface that works with existing Hadoop security
> components (authenticatication, confidentiality).
> Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141
> and HADOOP-10177.
> Hadoop KMS will provide an additional implementation of the Hadoop
> KeyProvider class. This implementation will be a client-server implementation.
> The client-server protocol will be secure:
> * Kerberos HTTP SPNEGO (authentication)
> * HTTPS for transport (confidentiality and integrity)
> * Hadoop ACLs (authorization)
> The Hadoop KMS implementation will not provide additional ACL to access
> encrypted files. For sophisticated access control requirements, HDFS ACLs
> (HDFS-4685) should be used.
> Basic key administration will be supported by the Hadoop KMS via the, already
> available, Hadoop KeyShell command line tool
> There are minor changes that must be done in Hadoop KeyProvider functionality:
> The KeyProvider contract, and the existing implementations, must be
> thread-safe
> KeyProvider API should have an API to generate the key material internally
> JavaKeyStoreProvider should use, if present, a password provided via
> configuration
> KeyProvider Option and Metadata should include a label (for easier
> cross-referencing)
> To avoid overloading the underlying KeyProvider implementation, the Hadoop
> KMS will cache keys using a TTL policy.
> Scalability and High Availability of the Hadoop KMS can achieved by running
> multiple instances behind a VIP/Load-Balancer. For High Availability, the
> underlying KeyProvider implementation used by the Hadoop KMS must be High
> Available.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987185#comment-13987185 ] Vinay Shukla commented on HADOOP-10433: --- [~tucu00] What is the audit story about the Key? Do we record, who did various key operations? I couldn't find it in the attached PDFs. > Key Management Server based on KeyProvider API > -- > > Key: HADOOP-10433 > URL: https://issues.apache.org/jira/browse/HADOOP-10433 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HadoopKMSDocsv2.pdf, KMS-doc.pdf > > > (from HDFS-6134 proposal) > Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying > KMS. It provides an interface that works with existing Hadoop security > components (authenticatication, confidentiality). > Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 > and HADOOP-10177. > Hadoop KMS will provide an additional implementation of the Hadoop > KeyProvider class. This implementation will be a client-server implementation. > The client-server protocol will be secure: > * Kerberos HTTP SPNEGO (authentication) > * HTTPS for transport (confidentiality and integrity) > * Hadoop ACLs (authorization) > The Hadoop KMS implementation will not provide additional ACL to access > encrypted files. For sophisticated access control requirements, HDFS ACLs > (HDFS-4685) should be used. > Basic key administration will be supported by the Hadoop KMS via the, already > available, Hadoop KeyShell command line tool > There are minor changes that must be done in Hadoop KeyProvider functionality: > The KeyProvider contract, and the existing implementations, must be > thread-safe > KeyProvider API should have an API to generate the key material internally > JavaKeyStoreProvider should use, if present, a password provided via > configuration > KeyProvider Option and Metadata should include a label (for easier > cross-referencing) > To avoid overloading the underlying KeyProvider implementation, the Hadoop > KMS will cache keys using a TTL policy. > Scalability and High Availability of the Hadoop KMS can achieved by running > multiple instances behind a VIP/Load-Balancer. For High Availability, the > underlying KeyProvider implementation used by the Hadoop KMS must be High > Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987191#comment-13987191 ] Alejandro Abdelnur commented on HADOOP-10433: - Vinay, there is a kms-audit.log file that logs all requests, both successful and failed. > Key Management Server based on KeyProvider API > -- > > Key: HADOOP-10433 > URL: https://issues.apache.org/jira/browse/HADOOP-10433 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HadoopKMSDocsv2.pdf, KMS-doc.pdf > > > (from HDFS-6134 proposal) > Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying > KMS. It provides an interface that works with existing Hadoop security > components (authenticatication, confidentiality). > Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 > and HADOOP-10177. > Hadoop KMS will provide an additional implementation of the Hadoop > KeyProvider class. This implementation will be a client-server implementation. > The client-server protocol will be secure: > * Kerberos HTTP SPNEGO (authentication) > * HTTPS for transport (confidentiality and integrity) > * Hadoop ACLs (authorization) > The Hadoop KMS implementation will not provide additional ACL to access > encrypted files. For sophisticated access control requirements, HDFS ACLs > (HDFS-4685) should be used. > Basic key administration will be supported by the Hadoop KMS via the, already > available, Hadoop KeyShell command line tool > There are minor changes that must be done in Hadoop KeyProvider functionality: > The KeyProvider contract, and the existing implementations, must be > thread-safe > KeyProvider API should have an API to generate the key material internally > JavaKeyStoreProvider should use, if present, a password provided via > configuration > KeyProvider Option and Metadata should include a label (for easier > cross-referencing) > To avoid overloading the underlying KeyProvider implementation, the Hadoop > KMS will cache keys using a TTL policy. > Scalability and High Availability of the Hadoop KMS can achieved by running > multiple instances behind a VIP/Load-Balancer. For High Availability, the > underlying KeyProvider implementation used by the Hadoop KMS must be High > Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987198#comment-13987198 ] Arpit Agarwal commented on HADOOP-10562: builds.apache.org appears to be down. > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.1.patch, HADOOP-10562.branch-1.1.patch, > HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10286) Allow RPCCallBenchmark to benchmark calls by different users
[ https://issues.apache.org/jira/browse/HADOOP-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10286: -- Attachment: HADOOP-10286.patch This patch enhances the existing RPCCallBenchmark to support testing variable loads from different users, as well as loading custom configurations, allowing it to test for different call queues and scenarios. > Allow RPCCallBenchmark to benchmark calls by different users > > > Key: HADOOP-10286 > URL: https://issues.apache.org/jira/browse/HADOOP-10286 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li > Attachments: HADOOP-10286.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10286) Allow RPCCallBenchmark to benchmark calls by different users
[ https://issues.apache.org/jira/browse/HADOOP-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10286: -- Attachment: HADOOP-10286.patch > Allow RPCCallBenchmark to benchmark calls by different users > > > Key: HADOOP-10286 > URL: https://issues.apache.org/jira/browse/HADOOP-10286 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li >Assignee: Chris Li > Attachments: HADOOP-10286.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10286) Allow RPCCallBenchmark to benchmark calls by different users
[ https://issues.apache.org/jira/browse/HADOOP-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10286: -- Attachment: (was: HADOOP-10286.patch) > Allow RPCCallBenchmark to benchmark calls by different users > > > Key: HADOOP-10286 > URL: https://issues.apache.org/jira/browse/HADOOP-10286 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li > Attachments: HADOOP-10286.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10286) Allow RPCCallBenchmark to benchmark calls by different users
[ https://issues.apache.org/jira/browse/HADOOP-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10286: -- Assignee: Chris Li Status: Patch Available (was: Open) > Allow RPCCallBenchmark to benchmark calls by different users > > > Key: HADOOP-10286 > URL: https://issues.apache.org/jira/browse/HADOOP-10286 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li >Assignee: Chris Li > Attachments: HADOOP-10286.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10281) Create a scheduler, which assigns schedulables a priority level
[ https://issues.apache.org/jira/browse/HADOOP-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10281: -- Attachment: HADOOP-10281.patch The previous version had issues scaling due to the use of ConcurrentLinkedQueue, which has a non-constant time size() call. > Create a scheduler, which assigns schedulables a priority level > --- > > Key: HADOOP-10281 > URL: https://issues.apache.org/jira/browse/HADOOP-10281 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li > Attachments: HADOOP-10281.patch, HADOOP-10281.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10563) Remove the dependency of jsp in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987243#comment-13987243
]
Hadoop QA commented on HADOOP-10563:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642915/HADOOP-10563.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3881//console
This message is automatically generated.
> Remove the dependency of jsp in trunk
> -
>
> Key: HADOOP-10563
> URL: https://issues.apache.org/jira/browse/HADOOP-10563
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HADOOP-10563.000.patch
>
>
> After HDFS-6252 neither hdfs nor yarn uses jsp, thus the dependency of the
> jsp can be removed from the pom.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10447) Implement C code for parsing Hadoop / HDFS URIs
[
https://issues.apache.org/jira/browse/HADOOP-10447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987258#comment-13987258
]
Colin Patrick McCabe commented on HADOOP-10447:
---
I think what we ought to do is use the native client for the {{hdfs://}}
scheme, and use the (existing) JNI client for the other schemes. We might also
implement a {{file://}} scheme in the long term.
> Implement C code for parsing Hadoop / HDFS URIs
> ---
>
> Key: HADOOP-10447
> URL: https://issues.apache.org/jira/browse/HADOOP-10447
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Colin Patrick McCabe
>
> We need some glue code for parsing Hadoop or HDFS URIs in C. Probably we
> should just put a small 'Path' wrapper around a URI parsing library like
> http://uriparser.sourceforge.net/ (BSD licensed)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987266#comment-13987266
]
Hadoop QA commented on HADOOP-10560:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642913/HADOOP-10560.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverControllerStress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3882//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3882//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3882//console
This message is automatically generated.
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
>
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Labels: performance
> Attachments: HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987270#comment-13987270
]
Hadoop QA commented on HADOOP-10562:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642944/HADOOP-10562.branch-1.1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3883//console
This message is automatically generated.
> Namenode exits on exception without printing stack trace in
> AbstractDelegationTokenSecretManager
>
>
> Key: HADOOP-10562
> URL: https://issues.apache.org/jira/browse/HADOOP-10562
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 1.2.1
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Critical
> Attachments: HADOOP-10562.1.patch, HADOOP-10562.branch-1.1.patch,
> HADOOP-10562.patch
>
>
> Not printing the stack trace makes debugging harder.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987286#comment-13987286 ] Uma Maheswara Rao G commented on HADOOP-10561: -- Once the branch merged, we can file one JIRS in MR for handling distcp case > Copy command with preserve option should handle Xattrs > -- > > Key: HADOOP-10561 > URL: https://issues.apache.org/jira/browse/HADOOP-10561 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > > The design docs for Xattrs stated that we handle preserve options with copy > commands > From doc: > Preserve option of commands like “cp -p” shell command and “distcp -p” should > work on XAttrs. > In the case of source fs supports XAttrs but target fs does not support, > XAttrs will be ignored > with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10433: Attachment: HADOOP-10433.patch new patch addressing Owen's suggestions. > Key Management Server based on KeyProvider API > -- > > Key: HADOOP-10433 > URL: https://issues.apache.org/jira/browse/HADOOP-10433 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf > > > (from HDFS-6134 proposal) > Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying > KMS. It provides an interface that works with existing Hadoop security > components (authenticatication, confidentiality). > Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 > and HADOOP-10177. > Hadoop KMS will provide an additional implementation of the Hadoop > KeyProvider class. This implementation will be a client-server implementation. > The client-server protocol will be secure: > * Kerberos HTTP SPNEGO (authentication) > * HTTPS for transport (confidentiality and integrity) > * Hadoop ACLs (authorization) > The Hadoop KMS implementation will not provide additional ACL to access > encrypted files. For sophisticated access control requirements, HDFS ACLs > (HDFS-4685) should be used. > Basic key administration will be supported by the Hadoop KMS via the, already > available, Hadoop KeyShell command line tool > There are minor changes that must be done in Hadoop KeyProvider functionality: > The KeyProvider contract, and the existing implementations, must be > thread-safe > KeyProvider API should have an API to generate the key material internally > JavaKeyStoreProvider should use, if present, a password provided via > configuration > KeyProvider Option and Metadata should include a label (for easier > cross-referencing) > To avoid overloading the underlying KeyProvider implementation, the Hadoop > KMS will cache keys using a TTL policy. > Scalability and High Availability of the Hadoop KMS can achieved by running > multiple instances behind a VIP/Load-Balancer. For High Availability, the > underlying KeyProvider implementation used by the Hadoop KMS must be High > Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10307) Support multiple Authentication mechanisms for HTTP
[ https://issues.apache.org/jira/browse/HADOOP-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987311#comment-13987311 ] Alejandro Abdelnur commented on HADOOP-10307: - Isn't this already supported by HADOOP-9054? > Support multiple Authentication mechanisms for HTTP > --- > > Key: HADOOP-10307 > URL: https://issues.apache.org/jira/browse/HADOOP-10307 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10307.patch > > > Currently it is possible to specify a custom Authentication Handler for HTTP > authentication. > We have a requirement to support multiple mechanisms to authenticate HTTP > access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987331#comment-13987331 ] Hudson commented on HADOOP-10562: - SUCCESS: Integrated in Hadoop-trunk-Commit #5590 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5590/]) HADOOP-10562. Undo previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1591785) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java HADOOP-10562. Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager. (Contributed by Suresh Srinivas) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1591782) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.1.patch, HADOOP-10562.branch-1.1.patch, > HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10562: --- Attachment: (was: HADOOP-10562.branch-1.1.patch) > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 1.2.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Attachments: HADOOP-10562.1.patch, HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987339#comment-13987339 ] Vinay Shukla commented on HADOOP-10433: --- Is there a sample of the audit log record? What field are audited? Would it be useful to have a common audit format across Hadoop? > Key Management Server based on KeyProvider API > -- > > Key: HADOOP-10433 > URL: https://issues.apache.org/jira/browse/HADOOP-10433 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, > HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf > > > (from HDFS-6134 proposal) > Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying > KMS. It provides an interface that works with existing Hadoop security > components (authenticatication, confidentiality). > Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 > and HADOOP-10177. > Hadoop KMS will provide an additional implementation of the Hadoop > KeyProvider class. This implementation will be a client-server implementation. > The client-server protocol will be secure: > * Kerberos HTTP SPNEGO (authentication) > * HTTPS for transport (confidentiality and integrity) > * Hadoop ACLs (authorization) > The Hadoop KMS implementation will not provide additional ACL to access > encrypted files. For sophisticated access control requirements, HDFS ACLs > (HDFS-4685) should be used. > Basic key administration will be supported by the Hadoop KMS via the, already > available, Hadoop KeyShell command line tool > There are minor changes that must be done in Hadoop KeyProvider functionality: > The KeyProvider contract, and the existing implementations, must be > thread-safe > KeyProvider API should have an API to generate the key material internally > JavaKeyStoreProvider should use, if present, a password provided via > configuration > KeyProvider Option and Metadata should include a label (for easier > cross-referencing) > To avoid overloading the underlying KeyProvider implementation, the Hadoop > KMS will cache keys using a TTL policy. > Scalability and High Availability of the Hadoop KMS can achieved by running > multiple instances behind a VIP/Load-Balancer. For High Availability, the > underlying KeyProvider implementation used by the Hadoop KMS must be High > Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HADOOP-10557: -- Assignee: Akira AJISAKA > FsShell -cp -p does not preserve extended ACLs > -- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > This issue tracks enhancing FsShell cp to > * preserve extended ACLs by -p option > or > * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10541) InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
[
https://issues.apache.org/jira/browse/HADOOP-10541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987356#comment-13987356
]
Swarnim Kulkarni commented on HADOOP-10541:
---
[~tedyu] If you are ok with it, I'll like to take a stab at this.
> InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
>
>
> Key: HADOOP-10541
> URL: https://issues.apache.org/jira/browse/HADOOP-10541
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> The same InputStream variable is used for minikdc.ldiff and minikdc-krb5.conf
> :
> {code}
> InputStream is = cl.getResourceAsStream("minikdc.ldiff");
> ...
> is = cl.getResourceAsStream("minikdc-krb5.conf");
> {code}
> Before the second assignment, is should be closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987358#comment-13987358
]
Hadoop QA commented on HADOOP-10433:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642989/HADOOP-10433.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-kms hadoop-dist:
org.apache.hadoop.ha.TestZKFailoverControllerStress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3885//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3885//console
This message is automatically generated.
> Key Management Server based on KeyProvider API
> --
>
> Key: HADOOP-10433
> URL: https://issues.apache.org/jira/browse/HADOOP-10433
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch,
> HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf
>
>
> (from HDFS-6134 proposal)
> Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying
> KMS. It provides an interface that works with existing Hadoop security
> components (authenticatication, confidentiality).
> Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141
> and HADOOP-10177.
> Hadoop KMS will provide an additional implementation of the Hadoop
> KeyProvider class. This implementation will be a client-server implementation.
> The client-server protocol will be secure:
> * Kerberos HTTP SPNEGO (authentication)
> * HTTPS for transport (confidentiality and integrity)
> * Hadoop ACLs (authorization)
> The Hadoop KMS implementation will not provide additional ACL to access
> encrypted files. For sophisticated access control requirements, HDFS ACLs
> (HDFS-4685) should be used.
> Basic key administration will be supported by the Hadoop KMS via the, already
> available, Hadoop KeyShell command line tool
> There are minor changes that must be done in Hadoop KeyProvider functionality:
> The KeyProvider contract, and the existing implementations, must be
> thread-safe
> KeyProvider API should have an API to generate the key material internally
> JavaKeyStoreProvider should use, if present, a password provided via
> configuration
> KeyProvider Option and Metadata should include a label (for easier
> cross-referencing)
> To avoid overloading the underlying KeyProvider implementation, the Hadoop
> KMS will cache keys using a TTL policy.
> Scalability and High Availability of the Hadoop KMS can achieved by running
> multiple instances behind a VIP/Load-Balancer. For High Availability, the
> underlying KeyProvider implementation used by the Hadoop KMS must be High
> Available.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987359#comment-13987359
]
Hadoop QA commented on HADOOP-10562:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642917/HADOOP-10562.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3886//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3886//console
This message is automatically generated.
> Namenode exits on exception without printing stack trace in
> AbstractDelegationTokenSecretManager
>
>
> Key: HADOOP-10562
> URL: https://issues.apache.org/jira/browse/HADOOP-10562
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 1.2.1
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Critical
> Attachments: HADOOP-10562.1.patch, HADOOP-10562.patch
>
>
> Not printing the stack trace makes debugging harder.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10541) InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
[
https://issues.apache.org/jira/browse/HADOOP-10541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HADOOP-10541:
--
Attachment: HADOOP-10541.1.patch.txt
Patch attached.
> InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
>
>
> Key: HADOOP-10541
> URL: https://issues.apache.org/jira/browse/HADOOP-10541
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-10541.1.patch.txt
>
>
> The same InputStream variable is used for minikdc.ldiff and minikdc-krb5.conf
> :
> {code}
> InputStream is = cl.getResourceAsStream("minikdc.ldiff");
> ...
> is = cl.getResourceAsStream("minikdc-krb5.conf");
> {code}
> Before the second assignment, is should be closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10541) InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
[
https://issues.apache.org/jira/browse/HADOOP-10541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HADOOP-10541:
--
Status: Patch Available (was: Open)
> InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
>
>
> Key: HADOOP-10541
> URL: https://issues.apache.org/jira/browse/HADOOP-10541
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-10541.1.patch.txt
>
>
> The same InputStream variable is used for minikdc.ldiff and minikdc-krb5.conf
> :
> {code}
> InputStream is = cl.getResourceAsStream("minikdc.ldiff");
> ...
> is = cl.getResourceAsStream("minikdc-krb5.conf");
> {code}
> Before the second assignment, is should be closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987361#comment-13987361
]
Hadoop QA commented on HADOOP-10562:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642917/HADOOP-10562.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3887//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3887//console
This message is automatically generated.
> Namenode exits on exception without printing stack trace in
> AbstractDelegationTokenSecretManager
>
>
> Key: HADOOP-10562
> URL: https://issues.apache.org/jira/browse/HADOOP-10562
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 1.2.1
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Critical
> Attachments: HADOOP-10562.1.patch, HADOOP-10562.patch
>
>
> Not printing the stack trace makes debugging harder.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
Benoy Antony created HADOOP-10565: - Summary: Support IP ranges (CIDR) in proxyuser.hosts Key: HADOOP-10565 URL: https://issues.apache.org/jira/browse/HADOOP-10565 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Benoy Antony Assignee: Benoy Antony In some use cases, there will be many hosts from which the user can impersonate. This requires specifying many ips in the XML configuration. It is cumbersome to specify and maintain long list of ips in proxyuser.hosts The problem can be solved if proxyuser.hosts accept ip ranges in CIDR format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10517) InputStream is not closed in two methods of JarFinder
[
https://issues.apache.org/jira/browse/HADOOP-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HADOOP-10517:
--
Status: Patch Available (was: Open)
> InputStream is not closed in two methods of JarFinder
> -
>
> Key: HADOOP-10517
> URL: https://issues.apache.org/jira/browse/HADOOP-10517
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-10517.1.patch.txt
>
>
> JarFinder#jarDir() and JarFinder#zipDir() have such code:
> {code}
> InputStream is = new FileInputStream(f);
> copyToZipStream(is, anEntry, zos);
> {code}
> The InputStream is not closed after copy operation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10517) InputStream is not closed in two methods of JarFinder
[
https://issues.apache.org/jira/browse/HADOOP-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HADOOP-10517:
--
Attachment: HADOOP-10517.1.patch.txt
Patch attached.
> InputStream is not closed in two methods of JarFinder
> -
>
> Key: HADOOP-10517
> URL: https://issues.apache.org/jira/browse/HADOOP-10517
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-10517.1.patch.txt
>
>
> JarFinder#jarDir() and JarFinder#zipDir() have such code:
> {code}
> InputStream is = new FileInputStream(f);
> copyToZipStream(is, anEntry, zos);
> {code}
> The InputStream is not closed after copy operation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10541) InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
[
https://issues.apache.org/jira/browse/HADOOP-10541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987421#comment-13987421
]
Hadoop QA commented on HADOOP-10541:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642996/HADOOP-10541.1.patch.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-minikdc.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3888//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3888//console
This message is automatically generated.
> InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
>
>
> Key: HADOOP-10541
> URL: https://issues.apache.org/jira/browse/HADOOP-10541
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-10541.1.patch.txt
>
>
> The same InputStream variable is used for minikdc.ldiff and minikdc-krb5.conf
> :
> {code}
> InputStream is = cl.getResourceAsStream("minikdc.ldiff");
> ...
> is = cl.getResourceAsStream("minikdc-krb5.conf");
> {code}
> Before the second assignment, is should be closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10563) Remove the dependency of jsp in trunk
[ https://issues.apache.org/jira/browse/HADOOP-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HADOOP-10563: Attachment: HADOOP-10563.001.patch > Remove the dependency of jsp in trunk > - > > Key: HADOOP-10563 > URL: https://issues.apache.org/jira/browse/HADOOP-10563 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HADOOP-10563.000.patch, HADOOP-10563.001.patch > > > After HDFS-6252 neither hdfs nor yarn uses jsp, thus the dependency of the > jsp can be removed from the pom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10565: -- Attachment: HADOOP-10565.patch Attaching patch which enhances proxyuser.hosts to accept Ip ranges in CIDR format. The patch uses a utility container MachineList to hold ips, ip ranges and/or host names. The current proxyuser authorize does a linear scan of ip list. MachineList does a HashSet Lookup. The performance test : Scenario 1: With 128 ip entries. Lookup a random ip 1 million times current implementation : 86940 ms (close to 0.1 ms for each authorize() call) New implementation using list of ips : 3420 ms New implementation using ip range : 3936 ms So with 128 entries , There is a 25 fold improvement in speed. Scenario 2 : With 4 ip entries. Lookup a random ip 1 million times current implementation : 7117 ms New implementation using list of ips : 3380 ms New implementation using ip range : 3969 ms The performance Test code is added in TestProxyUsers If specifying hostnames, the current implementation loops through the list hosts and tries to resolve the ip. The new implementation attempts to resolve the incoming ip's hostname with the list of hostnames. If that doesn't match , it loops through the lis hostnames. The new implementation (MachineList) supports an argument requesting to resolve the ip addresses of specified hostnames during MachineList construction. > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if proxyuser.hosts accept ip ranges in CIDR format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
Benoy Antony created HADOOP-10566: - Summary: Refactor proxyservers out of ProxyUsers Key: HADOOP-10566 URL: https://issues.apache.org/jira/browse/HADOOP-10566 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Benoy Antony Assignee: Benoy Antony Hadoop-10498 added proxyservers feature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUser.authorize() and isproxyUser are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10566: -- Description: HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUser.authorize() and isproxyUser are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. was: Hadoop-10498 added proxyservers feature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUser.authorize() and isproxyUser are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > > HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUser.authorize() and isproxyUser are synchronized and hence share > the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10566: -- Attachment: HADOOP-10566.patch Attaching a patch which moves the proxyServer logic to ProxyServer. The refresh of the ProxyServers can still be triggered via _ProxyUsers.refreshSuperUserGroupsConfiguration_ The set of proxyservers in ProxyServers is safely published via a volatile variable and this removes the cost of thread safety via locks (synchronized). This is similar to the approach taken in HADOOP-10448 > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch > > > HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10566: -- Description: HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. was: HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUser.authorize() and isproxyUser are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch > > > HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10448) Support pluggable mechanism to specify proxy user settings
[ https://issues.apache.org/jira/browse/HADOOP-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987430#comment-13987430 ] Benoy Antony commented on HADOOP-10448: --- Attaching a newer patch which includes HADOOP-10566 and moves proxyserver feature out of _ImpersonationProvider_ interface > Support pluggable mechanism to specify proxy user settings > -- > > Key: HADOOP-10448 > URL: https://issues.apache.org/jira/browse/HADOOP-10448 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.3.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10448.patch, HADOOP-10448.patch, > HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch, > HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch > > > We have a requirement to support large number of superusers. (users who > impersonate as another user) > (http://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html) > Currently each superuser needs to be defined in the core-site.xml via > proxyuser settings. This will be cumbersome when there are 1000 entries. > It seems useful to have a pluggable mechanism to specify proxy user settings > with the current approach as the default. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10448) Support pluggable mechanism to specify proxy user settings
[ https://issues.apache.org/jira/browse/HADOOP-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10448: -- Attachment: HADOOP-10448.patch > Support pluggable mechanism to specify proxy user settings > -- > > Key: HADOOP-10448 > URL: https://issues.apache.org/jira/browse/HADOOP-10448 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.3.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10448.patch, HADOOP-10448.patch, > HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch, > HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch > > > We have a requirement to support large number of superusers. (users who > impersonate as another user) > (http://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html) > Currently each superuser needs to be defined in the core-site.xml via > proxyuser settings. This will be cumbersome when there are 1000 entries. > It seems useful to have a pluggable mechanism to specify proxy user settings > with the current approach as the default. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10565: -- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-10469 > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if proxyuser.hosts accept ip ranges in CIDR format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987431#comment-13987431 ] Benoy Antony commented on HADOOP-10565: --- This patch delegates the ProxyUsers's ip/host name handling to MachineList. MachineList does ip and host lookup faster. It also enables ability to specify ip ranges in CIDR format. > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if proxyuser.hosts accept ip ranges in CIDR format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10566: -- Description: HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. was: HADOOP-10498 added proxyserversfeature in ProxyUsers. It is beneficial to treat this as a separate feature since 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The cardinality is different. 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized and hence share the same lock and impacts performance. Since these are two separate features, it will be an improvement to keep them separate. It also enables one to fine-tune each feature independently. > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10566: -- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-10469 > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10565: -- Status: Patch Available (was: Open) > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if proxyuser.hosts accept ip ranges in CIDR format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10566: -- Status: Patch Available (was: Open) > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10335) An ip whilelist based implementation to resolve Sasl properties per connection
[ https://issues.apache.org/jira/browse/HADOOP-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987435#comment-13987435 ] Benoy Antony commented on HADOOP-10335: --- can make use of HADOOP-10565 > An ip whilelist based implementation to resolve Sasl properties per connection > -- > > Key: HADOOP-10335 > URL: https://issues.apache.org/jira/browse/HADOOP-10335 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10335.patch, HADOOP-10335.patch, HADOOP-10335.pdf > > > As noted in HADOOP-10221, it is sometimes required for a Hadoop Server to > communicate with some client over encrypted channel and with some other > clients over unencrypted channel. > Hadoop-10221 introduced an interface _SaslPropertiesResolver_ and the > changes required to plugin and use _SaslPropertiesResolver_ to identify the > SaslProperties to be used for a connection. > In this jira, an ip-whitelist based implementation of > _SaslPropertiesResolver_ is attempted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: HADOOP-9705.4.patch
Moved the duplicated code into private method.
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: HADOOP-9705.4.patch
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: (was: HADOOP-9705.4.patch)
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
