[jira] [Commented] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055859#comment-14055859
]
Akira AJISAKA commented on HADOOP-10804:
bq. execute 'svn upgrade' only once in each node
each *repository* would be correct.
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Priority: Blocker
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10799) Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055860#comment-14055860 ] Varun Vasudev commented on HADOOP-10799: [~tucu00] just curious - any particular reason for this or simply refactoring? > Refactor HTTP delegation token logic from httpfs into reusable code in > hadoop-common. > - > > Key: HADOOP-10799 > URL: https://issues.apache.org/jira/browse/HADOOP-10799 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10799.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055847#comment-14055847
]
Akira AJISAKA commented on HADOOP-10804:
Please feel free to take over if you have the permission, thanks.
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Priority: Blocker
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10804:
---
Attachment: (was: HADOOP-10804.patch)
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Blocker
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10804:
---
Assignee: (was: Akira AJISAKA)
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Priority: Blocker
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055845#comment-14055845
]
Akira AJISAKA commented on HADOOP-10804:
Rethinking this, it would be sufficient to execute 'svn upgrade' only once in
each node. I'll remove the patch.
In addition, I don't have the permission to configure Jenkins server, so
changing assignee.
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Blocker
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lance Ying LI updated HADOOP-10558: --- Labels: debian hadoop hadoop-3.0.0-SNAPSHOT mapreduce ubuntu (was: (unknown) debian hadoop hadoop-3.0.0-SNAPSHOT mapreduce ubuntu) > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: debian, hadoop, hadoop-3.0.0-SNAPSHOT, mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lance Ying LI updated HADOOP-10558: --- Labels: debian hadoop hadoop-3.0.0-SNAPSHOT mapreduce ubuntu (was: hadoop mapreduce ubuntu) > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: (unknown), debian, hadoop, hadoop-3.0.0-SNAPSHOT, > mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lance Ying LI updated HADOOP-10558: --- Labels: (unknown) debian hadoop hadoop-3.0.0-SNAPSHOT mapreduce ubuntu (was: (unknown) debian hadoop hadoop-3.0.0-SNAPSHOT host is: local mapreduce ubuntu) > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: (unknown), debian, hadoop, hadoop-3.0.0-SNAPSHOT, > mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lance Ying LI updated HADOOP-10558: --- Labels: (unknown) debian hadoop hadoop-3.0.0-SNAPSHOT host is: local mapreduce ubuntu (was: debian hadoop hadoop-3.0.0-SNAPSHOT mapreduce ubuntu) > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: (unknown), debian, hadoop, hadoop-3.0.0-SNAPSHOT, > mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055842#comment-14055842 ] Lance Ying LI commented on HADOOP-10558: [root@debian0][140709125041][/opt/hadoop-3.0.0-SNAPSHOT/sbin]cat /etc/hosts 202.45.128.160 debian0 202.45.128.161 debian1 202.45.128.162 debian2 202.45.128.163 debian3 202.45.128.164 debian4 202.45.128.169 controller 202.45.128.168 switch 202.45.128.171 d1 202.45.128.172 d2 202.45.128.173 d3 202.45.128.174 d4 202.45.128.160 localhost > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: hadoop, mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10558) java.net.UnknownHostException: Invalid host name: local host is: (unknown)
[ https://issues.apache.org/jira/browse/HADOOP-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055841#comment-14055841 ] Lance Ying LI commented on HADOOP-10558: Hi, I am facing the same problem, it seems 'MapReduce' can not figure out the 'localhost' address. Here are some details, 14/07/09 12:25:45 INFO mapreduce.Job: Job job_1404879176168_0003 failed with state FAILED due to: Application application_1404879176168_0003 failed 2 times due to Error launching appattempt_1404879176168_0003_02. Got exception: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "d1":10088; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:742) at org.apache.hadoop.ipc.Client$Connection.(Client.java:400) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1452) at org.apache.hadoop.ipc.Client.call(Client.java:1381) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy28.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.UnknownHostException ... 12 more . Failing the application. > java.net.UnknownHostException: Invalid host name: local host is: (unknown) > -- > > Key: HADOOP-10558 > URL: https://issues.apache.org/jira/browse/HADOOP-10558 > Project: Hadoop Common > Issue Type: Bug > Environment: two node cluster > ubuntu 12.04 LTS in both > Java version "1.7.0_25" > node 1: core i5 4 GB RAM > node 2: core i3 4 GB RAM >Reporter: Sami Abobala > Labels: hadoop, mapreduce, ubuntu > > I had this exception every time i try to run map-red job, I went to > http://wiki.apache.org/hadoop/UnknownHost > and tried every possible solution and still have the same result > Task Id : attempt_1398945803120_0001_m_04_0, Status : FAILED > Container launch failed for container_1398945803120_0001_01_06 : > java.lang.reflect.UndeclaredThrowableException > . > . > Caused by: com.google.protobuf.ServiceException: > java.net.UnknownHostException: Invalid host name: local host is: (unknown); > destination host is: ""fatima-HP-ProBook-4520s":8042; > java.net.UnknownHostException; For more details see: > http://wiki.apache.org/hadoop/UnknownHost > . > . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055832#comment-14055832 ] Dmitry Sivachenko commented on HADOOP-10797: Okay, I see. The point to use /bin/sh is that most of these shell scripts are very simple and do not require any bash-specific things. And since there are systems that do not ship bash by default, this would eliminate one extra dependency. But it is not a big deal if you prefer to stick with more heavyweight bash instead. > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10804:
---
Attachment: HADOOP-10804.patch
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Blocker
> Attachments: HADOOP-10804.patch
>
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
[
https://issues.apache.org/jira/browse/HADOOP-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10804:
---
Description:
Now Jenkins is failing with the following message:
{code}
==
==
Testing patch for HADOOP-10661.
==
==
svn: E155036: Please see the 'svn upgrade' command
svn: E155036: The working copy at
'/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
is too old (format 10) to work with client version '1.8.8 (r1568071)'
(expects format 31). You need to upgrade the working copy first.
{code}
https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
was:
Now Jenkins is failing with the following message:
{code}
==
==
Testing patch for HADOOP-10661.
==
==
svn: E155036: Please see the 'svn upgrade' command
svn: E155036: The working copy at
'/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
is too old (format 10) to work with client version '1.8.8 (r1568071)'
(expects format 31). You need to upgrade the working copy first.
{code}
> Jenkins is failing due to the upgrade of svn client
> ---
>
> Key: HADOOP-10804
> URL: https://issues.apache.org/jira/browse/HADOOP-10804
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Blocker
>
> Now Jenkins is failing with the following message:
> {code}
> ==
> ==
> Testing patch for HADOOP-10661.
> ==
> ==
> svn: E155036: Please see the 'svn upgrade' command
> svn: E155036: The working copy at
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
> is too old (format 10) to work with client version '1.8.8 (r1568071)'
> (expects format 31). You need to upgrade the working copy first.
> {code}
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4231/console
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10804) Jenkins is failing due to the upgrade of svn client
Akira AJISAKA created HADOOP-10804:
--
Summary: Jenkins is failing due to the upgrade of svn client
Key: HADOOP-10804
URL: https://issues.apache.org/jira/browse/HADOOP-10804
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Blocker
Now Jenkins is failing with the following message:
{code}
==
==
Testing patch for HADOOP-10661.
==
==
svn: E155036: Please see the 'svn upgrade' command
svn: E155036: The working copy at
'/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
is too old (format 10) to work with client version '1.8.8 (r1568071)'
(expects format 31). You need to upgrade the working copy first.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HADOOP-10803) Update OpensslCipher#getInstance to accept CipherSuite#name format.
[
https://issues.apache.org/jira/browse/HADOOP-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-10803 started by Yi Liu.
> Update OpensslCipher#getInstance to accept CipherSuite#name format.
> ---
>
> Key: HADOOP-10803
> URL: https://issues.apache.org/jira/browse/HADOOP-10803
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
> Attachments: HADOOP-10803.patch
>
>
> The name format of {{org.apache.hadoop.crypto.CipherSuite}} is the same as
> transformation of {{javax.crypto.Cipher#getInstance}}.
> Let's update the {{OpensslCipher#getInstance}} to accept same format, then we
> can get OpensslCipher instance using CipherSuite.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10803) Update OpensslCipher#getInstance to accept CipherSuite#name format.
[
https://issues.apache.org/jira/browse/HADOOP-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HADOOP-10803:
Attachment: HADOOP-10803.patch
The patch allow OpensslCipher#getInstance able to accept a CipherSuite name,
and is in line with {{javax.crypto.Cipher#getInstance}}.
> Update OpensslCipher#getInstance to accept CipherSuite#name format.
> ---
>
> Key: HADOOP-10803
> URL: https://issues.apache.org/jira/browse/HADOOP-10803
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
> Attachments: HADOOP-10803.patch
>
>
> The name format of {{org.apache.hadoop.crypto.CipherSuite}} is the same as
> transformation of {{javax.crypto.Cipher#getInstance}}.
> Let's update the {{OpensslCipher#getInstance}} to accept same format, then we
> can get OpensslCipher instance using CipherSuite.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055648#comment-14055648 ] Andrew Wang commented on HADOOP-10720: -- And a few more things that floated across my head: * Should use a ThreadFactoryBuilder to name the fillerThreads and make them daemons * Would be good to Precondition check that the config values are >=0 or >=1 as appropriate * We're getting to the point where we might want a Conf object like NNConf or DNConf, up to you > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.1.patch, HADOOP-10720.2.patch, > HADOOP-10720.3.patch, HADOOP-10720.4.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10779) Generalize DFS_PERMISSIONS_SUPERUSERGROUP_KEY for any HCFS
[
https://issues.apache.org/jira/browse/HADOOP-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055633#comment-14055633
]
jay vyas commented on HADOOP-10779:
---
I agree that at least a best practice should be defined, maybe [~cmccabe] is
right - but at first glance I think I only half heartedly agree with his
argument :). Here is why:
I don't think the actual value of defining a *supergroup* is predicated on the
way that you *become* the supergroup. Rather, its in the semantics of how the
supergroup concept is used in the broader ecosystem. In hadoop, the
*supergroup* is used to determine which users should be allowed to do certain
operations. From a hadoop interoperability and compatibility perspective,
surely the idea that certain groups of people have privileged abilities in a
file system is generic, wouldn't you say ?
Maybe the next step is to find 3 or 4 examples of how the hadoop ecosystem is
using the dfs.permissions.supergroup parameter... Either (1) they are only
using it in HDFS specific context or (2) they are using it for broader
semantics.Maybe, to colin's idea, its possible that some are misusing the
parameter to mean *more * than it actually is meant to mean, in which case
individual bugs be filed where those instances are found.
> Generalize DFS_PERMISSIONS_SUPERUSERGROUP_KEY for any HCFS
> --
>
> Key: HADOOP-10779
> URL: https://issues.apache.org/jira/browse/HADOOP-10779
> Project: Hadoop Common
> Issue Type: Wish
> Components: fs
>Reporter: Martin Bukatovic
>Priority: Minor
>
> HDFS has configuration option {{dfs.permissions.superusergroup}} stored in
> {{hdfs-site.xml}} configuration file:
> {noformat}
>
> dfs.permissions.superusergroup
> supergroup
> The name of the group of super-users.
>
> {noformat}
> Since we have an option to use alternative Hadoop filesystems (HCFS), there is
> a question how to specify a supergroup in such case.
> Eg. would introducing HCFS option in say {{core-site.xml}} for this as shown
> below make sense?
> {noformat}
>
> hcfs.permissions.superusergroup
> ${dfs.permissions.superusergroup}
> The name of the group of super-users.
>
> {noformat}
> Or would you solve it in different way? I would like to at least declare
> a recommended approach for alternative Hadoop filesystems to follow.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10803) Update OpensslCipher#getInstance to accept CipherSuite#name format.
[
https://issues.apache.org/jira/browse/HADOOP-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HADOOP-10803:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-10150
> Update OpensslCipher#getInstance to accept CipherSuite#name format.
> ---
>
> Key: HADOOP-10803
> URL: https://issues.apache.org/jira/browse/HADOOP-10803
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
>
> The name format of {{org.apache.hadoop.crypto.CipherSuite}} is the same as
> transformation of {{javax.crypto.Cipher#getInstance}}.
> Let's update the {{OpensslCipher#getInstance}} to accept same format, then we
> can get OpensslCipher instance using CipherSuite.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Moved] (HADOOP-10803) Update OpensslCipher#getInstance to accept CipherSuite#name format.
[
https://issues.apache.org/jira/browse/HADOOP-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu moved HDFS-6644 to HADOOP-10803:
---
Component/s: (was: security)
security
Fix Version/s: (was: fs-encryption (HADOOP-10150 and HDFS-6134))
fs-encryption (HADOOP-10150 and HDFS-6134)
Target Version/s: (was: fs-encryption (HADOOP-10150 and HDFS-6134))
Affects Version/s: (was: fs-encryption (HADOOP-10150 and HDFS-6134))
fs-encryption (HADOOP-10150 and HDFS-6134)
Key: HADOOP-10803 (was: HDFS-6644)
Project: Hadoop Common (was: Hadoop HDFS)
> Update OpensslCipher#getInstance to accept CipherSuite#name format.
> ---
>
> Key: HADOOP-10803
> URL: https://issues.apache.org/jira/browse/HADOOP-10803
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
>
> The name format of {{org.apache.hadoop.crypto.CipherSuite}} is the same as
> transformation of {{javax.crypto.Cipher#getInstance}}.
> Let's update the {{OpensslCipher#getInstance}} to accept same format, then we
> can get OpensslCipher instance using CipherSuite.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1407#comment-1407 ] Andrew Wang commented on HADOOP-10720: -- One more thing I just thought of: it'd be nice to check that the client-side timeout works for these new methods too. Ideally we see get some kind of exception saying that we couldn't reach the KMS. > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.1.patch, HADOOP-10720.2.patch, > HADOOP-10720.3.patch, HADOOP-10720.4.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10725) Implement listStatus and getFileInfo in the native client
[ https://issues.apache.org/jira/browse/HADOOP-10725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10725: -- Attachment: HADOOP-10725-pnative.003.patch new patch with update: * Don't change the existing behavior of bailing out with an error if the port specified with hdfsBuilderSetNameNodePort differs from the URI port > Implement listStatus and getFileInfo in the native client > - > > Key: HADOOP-10725 > URL: https://issues.apache.org/jira/browse/HADOOP-10725 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10725-pnative.001.patch, > HADOOP-10725-pnative.002.patch, HADOOP-10725-pnative.003.patch > > > Implement listStatus and getFileInfo in the native client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[
https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055543#comment-14055543
]
Andrew Wang commented on HADOOP-10720:
--
Hey Arun, did a quick skim of this patch, few comments:
* typos: "IOExeption", "atleast", "funciton", "Defalt"
* New config keys should be added/documented in core-default.xml. Can c+p the
reference to core-default.xml for the javadocs in CommonConfigurationKeys.java.
* Let's also put the time unit into the config key name, e.g. appending {{_MS}}
* Could we rename the config params to say "refill" rather than just "fill"?
* I'd prefer to use ArrayLists rather than LinkedLists if we already know the
size upfront. Should be more efficient.
* It looks like the keyQueueFiller could race with an ongoing refill, so it
refills above the desired size.
* Tucu might have already asked this (didn't quite follow the above
discussion), but we want to make sure that there isn't a race such that a
client triggers a refill, a bunch of others come in and take all the keys, and
then the first client blocks on {{keyQueue.take()}}. One solution is using
{{poll()}} instead, continuing to trigger refills while the return value is
null.
* It looks like client requests will only will refill a single key. It seems
like we should request a batch of keys, assuming that network overhead is the
dominant cost here. I don't know how costly key generation is, but filling up
all the way to the {{threshold}} or at least to the {{lowWatermark}} would be
simple policies.
* Would also prefer {{lowWatermark}} rather than {{lowWaterMark}}, since
"watermark" is a single word.
Thanks for working on this! I'll give a more thorough review with your next
patch rev.
> KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
> ---
>
> Key: HADOOP-10720
> URL: https://issues.apache.org/jira/browse/HADOOP-10720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
> Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch,
> COMBO.patch, HADOOP-10720.1.patch, HADOOP-10720.2.patch,
> HADOOP-10720.3.patch, HADOOP-10720.4.patch, HADOOP-10720.patch,
> HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch
>
>
> KMS client/server should implement support for generating encrypted keys and
> decrypting them via the REST API being introduced by HADOOP-10719.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055514#comment-14055514 ] Giridharan Kesavan commented on HADOOP-10797: - I was against using /bin/sh and not against /usr/bin/env bash > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
[
https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055505#comment-14055505
]
Aaron T. Myers commented on HADOOP-10641:
-
HDFS-6469 still has several unanswered or un-retracted objections, perhaps most
notably this one from Suresh:
{quote}
I am very uncomfortable about adding all this complexity into HDFS.
{quote}
and this one from Todd:
{quote}
Lastly, a fully usable solution would be available to the community at large,
whereas the design you're proposing seems like it will only be usably
implemented by a proprietary extension (I don't consider the ZK "reference
implementation" likely to actually work in a usable fashion).
{quote}
I personally share both of these concerns with Suresh and Todd, specifically
that this design seems overly-complex and does not add much (or perhaps any)
benefit over a simpler solution that builds on the current HA system in Hadoop,
and I'm concerned about baking in dependence on a proprietary 3rd party system
for HA capabilities.
My main point here is that this work (HADOOP-10641) will not be useful to
Hadoop except in the context of HDFS-6469. So, I'm not OK with committing this
to trunk, at least until there's general agreement that HDFS-6469 is a
reasonable design that we should move forward with in Hadoop. As it stands, I
don't think there is such agreement; I certainly have not been convinced of
this yet. I'm OK with you committing this to a development branch if you want
to try to make progress that way, though.
> Introduce Coordination Engine
> -
>
> Key: HADOOP-10641
> URL: https://issues.apache.org/jira/browse/HADOOP-10641
> Project: Hadoop Common
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Konstantin Shvachko
>Assignee: Plamen Jeliazkov
> Attachments: HADOOP-10641.patch, HADOOP-10641.patch,
> HADOOP-10641.patch, hadoop-coordination.patch
>
>
> Coordination Engine (CE) is a system, which allows to agree on a sequence of
> events in a distributed system. In order to be reliable CE should be
> distributed by itself.
> Coordination Engine can be based on different algorithms (paxos, raft, 2PC,
> zab) and have different implementations, depending on use cases, reliability,
> availability, and performance requirements.
> CE should have a common API, so that it could serve as a pluggable component
> in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and
> HBase (HBASE-10909).
> First implementation is proposed to be based on ZooKeeper.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055490#comment-14055490 ] Dmitry Sivachenko commented on HADOOP-10797: Consider this article: http://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10780) namenode throws java.lang.OutOfMemoryError upon DatanodeProtocol.versionRequest from datanode
[ https://issues.apache.org/jira/browse/HADOOP-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055488#comment-14055488 ] Colin Patrick McCabe commented on HADOOP-10780: --- this one looks very straightforward. +1. Will commit tomorrow if no further comments > namenode throws java.lang.OutOfMemoryError upon > DatanodeProtocol.versionRequest from datanode > - > > Key: HADOOP-10780 > URL: https://issues.apache.org/jira/browse/HADOOP-10780 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 > Environment: FreeBSD-10/stable > openjdk version "1.7.0_60" > OpenJDK Runtime Environment (build 1.7.0_60-b19) > OpenJDK 64-Bit Server VM (build 24.60-b09, mixed mode) >Reporter: Dmitry Sivachenko > Attachments: buf_sz.patch > > > I am trying hadoop-2.4.1 on FreeBSD-10/stable. > namenode starts up, but after first datanode contacts it, it throws an > exception. > All limits seem to be high enough: > % limits -a > Resource limits (current): > cputime infinity secs > filesize infinity kB > datasize 33554432 kB > stacksize 524288 kB > coredumpsize infinity kB > memoryuseinfinity kB > memorylocked infinity kB > maxprocesses 122778 > openfiles 14 > sbsize infinity bytes > vmemoryuse infinity kB > pseudo-terminals infinity > swapuse infinity kB > 14944 1 S0:06.59 /usr/local/openjdk7/bin/java -Dproc_namenode > -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop > -Dhadoop.log.file=hadoop-hdfs-namenode-nezabudka3-00.log > -Dhadoop.home.dir=/usr/local -Dhadoop.id.str=hdfs > -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml > -Djava.net.preferIPv4Stack=true -Xmx32768m -Xms32768m > -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m > -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m > -Djava.library.path=/usr/local/lib -Dhadoop.security.logger=INFO,RFAS > org.apache.hadoop.hdfs.server.namenode.NameNode > From the namenode's log: > 2014-07-03 23:28:15,070 WARN [IPC Server handler 5 on 8020] ipc.Server > (Server.java:run(2032)) - IPC Server handler 5 on 8020, call > org.apache.hadoop.hdfs.server.protocol.Datano > deProtocol.versionRequest from 5.255.231.209:57749 Call#842 Retry#0 > java.lang.OutOfMemoryError > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Native > Method) > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(JniBasedUnixGroupsMapping.java:80) > at > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50) > at org.apache.hadoop.security.Groups.getGroups(Groups.java:139) > at > org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1417) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:81) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3331) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkSuperuserPrivilege(FSNamesystem.java:5491) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.versionRequest(NameNodeRpcServer.java:1082) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.versionRequest(DatanodeProtocolServerSideTranslatorPB.java:234) > at > org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28069) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > I did not have such an issue with hadoop-1.2.1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055485#comment-14055485 ] Dmitry Sivachenko commented on HADOOP-10797: Debian is not the only system around. /bin/sh is a standard location for bourne shell. FreeBSD, for instance, installs bash as "/usr/local/bin/bash". That is why I propose to either use "/bin/sh" and do not rely on bash extensions in scripts, use good old Bourne shell syntax, or use "/usr/bin/env bash", which will invoke bash in a portable way and work on both debian and FreeBSD. > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10802) Add metrics for KMS client and server encrypted key caches
[ https://issues.apache.org/jira/browse/HADOOP-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055458#comment-14055458 ] Andrew Wang commented on HADOOP-10802: -- A few ideas for metrics (just suggestions): * # of times the cache is empty (0) when a request comes in * Moving average cache size * Some indication of peak load, e.g. max request rate over different time durations * A histogram showing how "hot" some keyIds are compared to others, so we know how skewed demand is > Add metrics for KMS client and server encrypted key caches > -- > > Key: HADOOP-10802 > URL: https://issues.apache.org/jira/browse/HADOOP-10802 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Arun Suresh > > HADOOP-10720 is adding KMS server and client caches for encrypted keys for > performance reasons. It would be good to add metrics to make sure that the > cache is working as expected, and to inform future dynamic cache sizing and > refilling policies. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055459#comment-14055459 ] Hudson commented on HADOOP-10781: - SUCCESS: Integrated in Hadoop-trunk-Commit #5845 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5845/]) move HADOOP-10781 to 2.6 (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1608936) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.6.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10802) Add metrics for KMS client and server encrypted key caches
Andrew Wang created HADOOP-10802: Summary: Add metrics for KMS client and server encrypted key caches Key: HADOOP-10802 URL: https://issues.apache.org/jira/browse/HADOOP-10802 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Arun Suresh HADOOP-10720 is adding KMS server and client caches for encrypted keys for performance reasons. It would be good to add metrics to make sure that the cache is working as expected, and to inform future dynamic cache sizing and refilling policies. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055434#comment-14055434 ] Colin Patrick McCabe commented on HADOOP-10781: --- bq. It looks like this was not merged down to branch-2.5, correct? If so, can we please change the target version and fix version, and also move attribution to the 2.6.0 section in CHANGES.txt? Correct. Thanks for spotting this; I updated the CHANGES.txt in trunk and branch-2 > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.6.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055438#comment-14055438 ] Giridharan Kesavan commented on HADOOP-10797: - /bin/sh is symlined to /bin/dash on debian platforms. So I think it makes sense to use /bin/bash and not /bin/sh. > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10614) CBZip2InputStream is not threadsafe
[
https://issues.apache.org/jira/browse/HADOOP-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055435#comment-14055435
]
Sandy Ryza commented on HADOOP-10614:
-
Naw these are typical
> CBZip2InputStream is not threadsafe
> ---
>
> Key: HADOOP-10614
> URL: https://issues.apache.org/jira/browse/HADOOP-10614
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Xiangrui Meng
>Assignee: Xiangrui Meng
> Fix For: 1.3.0, 2.5.0
>
> Attachments: bzip2-2.diff, bzip2.diff
>
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the
> implementation is not threadsafe. This is not a really problem for Hadoop
> MapReduce because Hadoop runs each task in a separate JVM. But for other
> libraries that utilize multithreading and use Hadoop's InputFormat, e.g.,
> Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
>
> org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
> java.io.InputStream.read(InputStream.java:101)
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169)
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176)
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43)
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198)
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181)
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000)
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
> org.apache.spark.scheduler.Task.run(Task.scala:51)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:724)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10781: -- Target Version/s: 2.6.0 (was: 2.5.0) Fix Version/s: (was: 2.5.0) 2.6.0 > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.6.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10794) A hadoop cluster needs clock synchronization
[ https://issues.apache.org/jira/browse/HADOOP-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055429#comment-14055429 ] Andrew Wang commented on HADOOP-10794: -- Building on what Steve said, is this something that would better belong in a monitoring tool like Ganglia or Ambari? Within Hadoop, I think logging a warning for egregious errors like a negative time is good, but I don't think we should do much more than that. > A hadoop cluster needs clock synchronization > > > Key: HADOOP-10794 > URL: https://issues.apache.org/jira/browse/HADOOP-10794 > Project: Hadoop Common > Issue Type: Bug >Reporter: Zhijie Shen > > As a distributed system, a hadoop cluster wants the clock on all the > participating hosts synchronized. Otherwise, some problems might happen. For > example, in YARN-2251, due to the clock on the host for the task container > falls behind that on the host of the AM container, the computed elapsed time > (the diff between the timestamps produced on two hosts) becomes negative. > In YARN-2251, we tried to mask the negative elapsed time. However, we should > seek for a decent long term solution, such as providing mechanism to do and > check clock synchronization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10614) CBZip2InputStream is not threadsafe
[
https://issues.apache.org/jira/browse/HADOOP-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055398#comment-14055398
]
Andrew Ash commented on HADOOP-10614:
-
I can't quite tell -- are these Hudson failures a problem?
> CBZip2InputStream is not threadsafe
> ---
>
> Key: HADOOP-10614
> URL: https://issues.apache.org/jira/browse/HADOOP-10614
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Xiangrui Meng
>Assignee: Xiangrui Meng
> Fix For: 1.3.0, 2.5.0
>
> Attachments: bzip2-2.diff, bzip2.diff
>
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the
> implementation is not threadsafe. This is not a really problem for Hadoop
> MapReduce because Hadoop runs each task in a separate JVM. But for other
> libraries that utilize multithreading and use Hadoop's InputFormat, e.g.,
> Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
>
> org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
> java.io.InputStream.read(InputStream.java:101)
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169)
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176)
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43)
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198)
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181)
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000)
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
> org.apache.spark.scheduler.Task.run(Task.scala:51)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:724)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10727) GraphiteSink metric names should not contain "."
[ https://issues.apache.org/jira/browse/HADOOP-10727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055394#comment-14055394 ] Babak Behzad commented on HADOOP-10727: --- Thanks Ravi. Both of your comments are incorporated in the new patch I uploaded. > GraphiteSink metric names should not contain "." > > > Key: HADOOP-10727 > URL: https://issues.apache.org/jira/browse/HADOOP-10727 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Babak Behzad >Priority: Minor > Attachments: HADOOP-10727-v2.patch, HADOOP-10727.patch > > > Sometimes the names of the metrics sent to Graphite contain "." in them (such > as hostnames). Graphite interpret these dots as a separator for directories > causing long directory hierarchies. It would be easier to replace them to "_" > in order to have easier to read metric names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10727) GraphiteSink metric names should not contain "."
[ https://issues.apache.org/jira/browse/HADOOP-10727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babak Behzad updated HADOOP-10727: -- Attachment: HADOOP-10727-v2.patch > GraphiteSink metric names should not contain "." > > > Key: HADOOP-10727 > URL: https://issues.apache.org/jira/browse/HADOOP-10727 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Babak Behzad >Priority: Minor > Attachments: HADOOP-10727-v2.patch, HADOOP-10727.patch > > > Sometimes the names of the metrics sent to Graphite contain "." in them (such > as hostnames). Graphite interpret these dots as a separator for directories > causing long directory hierarchies. It would be easier to replace them to "_" > in order to have easier to read metric names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
[ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055383#comment-14055383 ] Konstantin Boudnik commented on HADOOP-10641: - [~atm] HDFS-6469 doesn't have _a lot of objections_ really. It has some questions, but they are already pretty much covered, IMO, by [~shv]. So it doesn't looks like HDFS ticket is blocking the passage of this one. > Introduce Coordination Engine > - > > Key: HADOOP-10641 > URL: https://issues.apache.org/jira/browse/HADOOP-10641 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Konstantin Shvachko >Assignee: Plamen Jeliazkov > Attachments: HADOOP-10641.patch, HADOOP-10641.patch, > HADOOP-10641.patch, hadoop-coordination.patch > > > Coordination Engine (CE) is a system, which allows to agree on a sequence of > events in a distributed system. In order to be reliable CE should be > distributed by itself. > Coordination Engine can be based on different algorithms (paxos, raft, 2PC, > zab) and have different implementations, depending on use cases, reliability, > availability, and performance requirements. > CE should have a common API, so that it could serve as a pluggable component > in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and > HBase (HBASE-10909). > First implementation is proposed to be based on ZooKeeper. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055366#comment-14055366 ] Chris Nauroth commented on HADOOP-10781: Thanks, Colin. I'd like to wait for [~kihwal] to chime in first to confirm before we proceed. I wouldn't want to put in an ifdef that wasn't really needed. It looks like this was not merged down to branch-2.5, correct? If so, can we please change the target version and fix version, and also move attribution to the 2.6.0 section in CHANGES.txt? > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[
https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055316#comment-14055316
]
Colin Patrick McCabe commented on HADOOP-10781:
---
Thanks for that information, Chris. I guess the key question is whether
{{getgrouplist}} returning 0 is an error under Linux. This seems like an odd
design decision for Linux to make, since then there would be no way to indicate
that there were no groups associated with a user.
Still, if you want to revert this and then do a new patch where you have an
#ifdef BSD or something, I have no problem with that. It is OK for us to treat
a return value of 0 as an error on Linux, since our error handling strategy is
to assume there are no groups anyway.
> Unportable getgrouplist() usage breaks FreeBSD
> --
>
> Key: HADOOP-10781
> URL: https://issues.apache.org/jira/browse/HADOOP-10781
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Dmitry Sivachenko
>Assignee: Dmitry Sivachenko
> Fix For: 2.5.0
>
> Attachments: getgrouplist.patch
>
>
> getgrouplist() has different return values on Linux and FreeBSD:
> Linux: either the number of groups (positive) or -1 on error
> FreeBSD: 0 on success or -1 on error
> The return value of getgrouplist() is analyzed in Linux-specific way in
> hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c,
> in function hadoop_user_info_getgroups() which breaks FreeBSD.
> In this function you have 3 choices for the return value
> ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid,
> uinfo->gids, &ngroups);
> 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change
> this to ret >= 0
> 2) First condition is false and ret != -1: impossible according to manpage
> 3) ret == 1 -- OK for both Linux and FreeBSD
> So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd
> case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055311#comment-14055311 ] Chris Nauroth commented on HADOOP-10781: Unfortunately, we were dealing with a situation where the underlying library implementation had deviated from the man page. > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10727) GraphiteSink metric names should not contain "."
[
https://issues.apache.org/jira/browse/HADOOP-10727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055293#comment-14055293
]
Ravi Prakash commented on HADOOP-10727:
---
Hi Babak!
Thanks for the patch.
{code}if (escape_character == null)
escape_character = "_";{code}
please include a closing and ending brace. Or you could just use the ? :
operator ;
Also please write a comment on the regex, to help identify what it is filtering
for. I'm a +1 on the patch after those changes.
> GraphiteSink metric names should not contain "."
>
>
> Key: HADOOP-10727
> URL: https://issues.apache.org/jira/browse/HADOOP-10727
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Babak Behzad
>Priority: Minor
> Attachments: HADOOP-10727.patch
>
>
> Sometimes the names of the metrics sent to Graphite contain "." in them (such
> as hostnames). Graphite interpret these dots as a separator for directories
> causing long directory hierarchies. It would be easier to replace them to "_"
> in order to have easier to read metric names.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055276#comment-14055276 ] Dmitry Sivachenko commented on HADOOP-10781: According to the manual: If user does not refer to a valid user on the system, getgrouplist() shall return 0, and set the value referenced by ngroups to 0. But at that point we already checked that this is valid user (at the beginning of that function), so this should be valid user and getgrouplist() should return positive value. > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055257#comment-14055257 ] Hudson commented on HADOOP-10781: - SUCCESS: Integrated in Hadoop-trunk-Commit #5843 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5843/]) HADOOP-10781. Unportable getgrouplist usage breaks FreeBSD (Dmitry Sivachenko via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1608869) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[
https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055256#comment-14055256
]
Chris Nauroth commented on HADOOP-10781:
Oops, looks like I was too late. :-) No worries.
I wanted to bring up that this changes the error handling behavior on Linux
too, and I wonder if it has reintroduced the segfault that had been seen
earlier. I believe the problem is that we'd get a 0 return code from
{{getgrouplist}}, we'd treat that as success, and then that would drive us into
a {{realloc}} based on {{ngroups}}. However, {{ngroups}} could be 0 at that
point (or perhaps random uninitialized memory), and then we'd get the segfault.
Admittedly, this was happening with buggy implementations of nscd, but we
wanted to be conservative in the error handling.
[~kihwal], I know you had spent the most time looking at these problems. Do
you think we need to revert HADOOP-10781 and find a different way to do this
patch for FreeBSD?
> Unportable getgrouplist() usage breaks FreeBSD
> --
>
> Key: HADOOP-10781
> URL: https://issues.apache.org/jira/browse/HADOOP-10781
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Dmitry Sivachenko
>Assignee: Dmitry Sivachenko
> Fix For: 2.5.0
>
> Attachments: getgrouplist.patch
>
>
> getgrouplist() has different return values on Linux and FreeBSD:
> Linux: either the number of groups (positive) or -1 on error
> FreeBSD: 0 on success or -1 on error
> The return value of getgrouplist() is analyzed in Linux-specific way in
> hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c,
> in function hadoop_user_info_getgroups() which breaks FreeBSD.
> In this function you have 3 choices for the return value
> ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid,
> uinfo->gids, &ngroups);
> 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change
> this to ret >= 0
> 2) First condition is false and ret != -1: impossible according to manpage
> 3) ret == 1 -- OK for both Linux and FreeBSD
> So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd
> case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10661) Ineffective user/passsword check in FTPFileSystem#initialize()
[
https://issues.apache.org/jira/browse/HADOOP-10661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated HADOOP-10661:
-
Attachment: HADOOP-10661.patch
trigger QA
> Ineffective user/passsword check in FTPFileSystem#initialize()
> --
>
> Key: HADOOP-10661
> URL: https://issues.apache.org/jira/browse/HADOOP-10661
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
>Priority: Minor
> Attachments: HADOOP-10661.patch, HADOOP-10661.patch
>
>
> Here is related code:
> {code}
> userAndPassword = (conf.get("fs.ftp.user." + host, null) + ":" + conf
> .get("fs.ftp.password." + host, null));
> if (userAndPassword == null) {
> throw new IOException("Invalid user/passsword specified");
> }
> {code}
> The intention seems to be checking that username / password should not be
> null.
> But due to the presence of colon, the above check is not effective.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10781: -- Assignee: Dmitry Sivachenko > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055252#comment-14055252 ] Colin Patrick McCabe commented on HADOOP-10781: --- I'm sorry, Chris, I committed before I saw your message. Perhaps we can open a follow-up JIRA? > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko >Assignee: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10781: -- Resolution: Fixed Fix Version/s: 2.5.0 Target Version/s: 2.5.0 Status: Resolved (was: Patch Available) > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055245#comment-14055245 ] Chris Nauroth commented on HADOOP-10781: Can you please hold off committing? More comments forthcoming... > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Fix For: 2.5.0 > > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055225#comment-14055225 ] Colin Patrick McCabe commented on HADOOP-10781: --- +1. Thanks, Dmitry. > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10792) Add FileSystem#closeIfNotReferred method
[
https://issues.apache.org/jira/browse/HADOOP-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055209#comment-14055209
]
Chris Nauroth commented on HADOOP-10792:
This sounds like ideally you'd want {{FileSystem#close}} to do reference
counting. Unfortunately, there has been community objection to this in the
past. (See discussion in HADOOP-4655.) I suspect Steve's suggestion is your
best option for now.
> Add FileSystem#closeIfNotReferred method
>
>
> Key: HADOOP-10792
> URL: https://issues.apache.org/jira/browse/HADOOP-10792
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Kousuke Saruta
>
> FileSystem#close closes FileSystem even if the same instance of FileSystem is
> referred by someone.
> For instance, a library using FileSystem calls FileSystem.get, and a program
> using the library calls FileSystem.get, both of instances of FileSystem is
> same.
> When the library and the program is implemented as different threads and one
> calls FileSystem.close, another fails most of operations of FileSystem.
> So, we need the method like cloesIfNotReferred, which closes FileSystem only
> if a instance of FileSystem is not referred by anyone.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[
https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055192#comment-14055192
]
Alejandro Abdelnur commented on HADOOP-10720:
-
bq. Also, I felt having EncryptedKeyVersion in CryptoExtension makes more sense
since this will allow different implementations of CE to have its own
EncKeyVersion without forcing it to be a subclass of KPCE.
We are following this pattern already in other {{KeyProvider}} classes, so I
would go for requiring subclassing.
bq. {{getAtLeast()}} rename to {{getAtMost()}}
sure.
> KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
> ---
>
> Key: HADOOP-10720
> URL: https://issues.apache.org/jira/browse/HADOOP-10720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
> Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch,
> COMBO.patch, HADOOP-10720.1.patch, HADOOP-10720.2.patch,
> HADOOP-10720.3.patch, HADOOP-10720.4.patch, HADOOP-10720.patch,
> HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch
>
>
> KMS client/server should implement support for generating encrypted keys and
> decrypting them via the REST API being introduced by HADOOP-10719.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10800) Refactor HttpFS to use hadoop-common HTTP delegation token support.
[ https://issues.apache.org/jira/browse/HADOOP-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10800: Attachment: HADOOP-10800.patch patch requires HADOOP-10799 > Refactor HttpFS to use hadoop-common HTTP delegation token support. > --- > > Key: HADOOP-10800 > URL: https://issues.apache.org/jira/browse/HADOOP-10800 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Alejandro Abdelnur > Attachments: HADOOP-10800.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10801) Fix dead link in site.xml
[
https://issues.apache.org/jira/browse/HADOOP-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055163#comment-14055163
]
Arpit Agarwal commented on HADOOP-10801:
+1 pending Jenkins, built the site and verified links work.
> Fix dead link in site.xml
> -
>
> Key: HADOOP-10801
> URL: https://issues.apache.org/jira/browse/HADOOP-10801
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HADOOP-10801.patch
>
>
> Documents for FileSystem API definition were created in HADOOP-9361 but not
> linked.
> In hadoop-project/src/site/site.xml,
> {code}
>href="hadoop-project-dist/hadoop-common/index.html"/>
> {code}
> should be
> {code}
>href="hadoop-project-dist/hadoop-common/filesystem/index.html"/>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
[ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055156#comment-14055156 ] Aaron T. Myers commented on HADOOP-10641: - I don't think we should be committing this to the trunk of Hadoop Common until it's actually 100% clear that HDFS will have a dependency on it. As it stands, there are a lot of objections in HDFS-6469 about adding this to the NN at all. If you want to add it to a development branch in Hadoop then I won't stand in the way of that, but I honestly believe you'll be able to make a lot more headway more quickly if you go the route of putting this in a separate project. > Introduce Coordination Engine > - > > Key: HADOOP-10641 > URL: https://issues.apache.org/jira/browse/HADOOP-10641 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Konstantin Shvachko >Assignee: Plamen Jeliazkov > Attachments: HADOOP-10641.patch, HADOOP-10641.patch, > HADOOP-10641.patch, hadoop-coordination.patch > > > Coordination Engine (CE) is a system, which allows to agree on a sequence of > events in a distributed system. In order to be reliable CE should be > distributed by itself. > Coordination Engine can be based on different algorithms (paxos, raft, 2PC, > zab) and have different implementations, depending on use cases, reliability, > availability, and performance requirements. > CE should have a common API, so that it could serve as a pluggable component > in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and > HBase (HBASE-10909). > First implementation is proposed to be based on ZooKeeper. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10801) Fix dead link in site.xml
[
https://issues.apache.org/jira/browse/HADOOP-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10801:
---
Attachment: HADOOP-10801.patch
Attaching a patch.
> Fix dead link in site.xml
> -
>
> Key: HADOOP-10801
> URL: https://issues.apache.org/jira/browse/HADOOP-10801
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HADOOP-10801.patch
>
>
> Documents for FileSystem API definition were created in HADOOP-9361 but not
> linked.
> In hadoop-project/src/site/site.xml,
> {code}
>href="hadoop-project-dist/hadoop-common/index.html"/>
> {code}
> should be
> {code}
>href="hadoop-project-dist/hadoop-common/filesystem/index.html"/>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10801) Fix dead link in site.xml
[
https://issues.apache.org/jira/browse/HADOOP-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10801:
---
Status: Patch Available (was: Open)
> Fix dead link in site.xml
> -
>
> Key: HADOOP-10801
> URL: https://issues.apache.org/jira/browse/HADOOP-10801
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HADOOP-10801.patch
>
>
> Documents for FileSystem API definition were created in HADOOP-9361 but not
> linked.
> In hadoop-project/src/site/site.xml,
> {code}
>href="hadoop-project-dist/hadoop-common/index.html"/>
> {code}
> should be
> {code}
>href="hadoop-project-dist/hadoop-common/filesystem/index.html"/>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10801) Fix dead link in site.xml
Akira AJISAKA created HADOOP-10801:
--
Summary: Fix dead link in site.xml
Key: HADOOP-10801
URL: https://issues.apache.org/jira/browse/HADOOP-10801
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.5.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Documents for FileSystem API definition were created in HADOOP-9361 but not
linked.
In hadoop-project/src/site/site.xml,
{code}
{code}
should be
{code}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10641) Introduce Coordination Engine
[ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HADOOP-10641: -- Status: Patch Available (was: In Progress) > Introduce Coordination Engine > - > > Key: HADOOP-10641 > URL: https://issues.apache.org/jira/browse/HADOOP-10641 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Konstantin Shvachko >Assignee: Plamen Jeliazkov > Attachments: HADOOP-10641.patch, HADOOP-10641.patch, > HADOOP-10641.patch, hadoop-coordination.patch > > > Coordination Engine (CE) is a system, which allows to agree on a sequence of > events in a distributed system. In order to be reliable CE should be > distributed by itself. > Coordination Engine can be based on different algorithms (paxos, raft, 2PC, > zab) and have different implementations, depending on use cases, reliability, > availability, and performance requirements. > CE should have a common API, so that it could serve as a pluggable component > in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and > HBase (HBASE-10909). > First implementation is proposed to be based on ZooKeeper. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[
https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055134#comment-14055134
]
Arun Suresh commented on HADOOP-10720:
--
[~tucu00], thanks for the feedback. Will upload a patch shortly addressing
these. Couplo things though :
bq. why are we moving EncryptedKeyVersion inside of CryptoExtension?
So the {{KMSClientProvider}} is an instance of {{CryptoExtension}}, not
{{KeyProviderCryptoExtension}}. Thus since the {{EncryptedKeyVersion}}
constructor is protected, it will not be assessable to {{KMSClientProvider}} to
subclass unless it part {{CryptoExtension}}.
Also, I felt having {{EncryptedKeyVersion}} in {{CryptoExtension}} makes more
sense since this will allow different implementations of CE to have its own
EncKeyVersion without forcing it to be a subclass of KPCE.
bq. getAtLeast(), if queue is empty should trigger async queue filling and fill
1 value synchronous to avoid stealing from other request.
Sure, but I put the getAtLeast() to enforce the contract that the client
requires at-least 'n' keys from the call.. Maybe I could change the signature
to getAtMost() ? and return 1 if Queue is empty ?
> KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
> ---
>
> Key: HADOOP-10720
> URL: https://issues.apache.org/jira/browse/HADOOP-10720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
> Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch,
> COMBO.patch, HADOOP-10720.1.patch, HADOOP-10720.2.patch,
> HADOOP-10720.3.patch, HADOOP-10720.4.patch, HADOOP-10720.patch,
> HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch
>
>
> KMS client/server should implement support for generating encrypted keys and
> decrypting them via the REST API being introduced by HADOOP-10719.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10799) Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10799: Status: Patch Available (was: Open) > Refactor HTTP delegation token logic from httpfs into reusable code in > hadoop-common. > - > > Key: HADOOP-10799 > URL: https://issues.apache.org/jira/browse/HADOOP-10799 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10799.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10799) Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10799: Attachment: HADOOP-10799.patch The code in this patch is 99% from httpfs, with minor changes to make it reusable. we cannot have it in hadoop-auth because all the dependencies from common (configuration, delegationtokens, etc). HADOOP-10800 will remove the DT code from httpfs and use this one. > Refactor HTTP delegation token logic from httpfs into reusable code in > hadoop-common. > - > > Key: HADOOP-10799 > URL: https://issues.apache.org/jira/browse/HADOOP-10799 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10799.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10793) Key Shell args use double dash style
[ https://issues.apache.org/jira/browse/HADOOP-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055127#comment-14055127 ] Mike Yoder commented on HADOOP-10793: - Hi [~lmccay] - to be clear, I like the double dash style, too - but I have to agree with others that being consistent with other existing tools trumps that. The single dash might be slightly archaic, but it's better than different argument schemes for different tools. This ecosystem is complicated enough without that. :-) I would imagine that the change would be to simply lop off one dash. As for the arg parsing, it's not _bad_ as is, but if there is a package that was built to handle command line args we should at least have a look at it. > Key Shell args use double dash style > > > Key: HADOOP-10793 > URL: https://issues.apache.org/jira/browse/HADOOP-10793 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder > > Follow-on from HADOOP-10736 as per [~andrew.wang] - the key shell uses the > gnu double dash style for command line args, while other command line > programs use a single dash. Consider changing this, and consider another > argument parsing scheme, like the CommandLine class. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10799) Refactor HTTP DT logic from httpfs into reusable code in hadoop-common.
Alejandro Abdelnur created HADOOP-10799: --- Summary: Refactor HTTP DT logic from httpfs into reusable code in hadoop-common. Key: HADOOP-10799 URL: https://issues.apache.org/jira/browse/HADOOP-10799 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10799) Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10799: Summary: Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common. (was: Refactor HTTP DT logic from httpfs into reusable code in hadoop-common.) > Refactor HTTP delegation token logic from httpfs into reusable code in > hadoop-common. > - > > Key: HADOOP-10799 > URL: https://issues.apache.org/jira/browse/HADOOP-10799 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10800) Refactor HttpFS to use hadoop-common HTTP delegation token support.
Alejandro Abdelnur created HADOOP-10800: --- Summary: Refactor HttpFS to use hadoop-common HTTP delegation token support. Key: HADOOP-10800 URL: https://issues.apache.org/jira/browse/HADOOP-10800 Project: Hadoop Common Issue Type: Sub-task Reporter: Alejandro Abdelnur -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10771) Refactor HTTP delegation support out of httpfs to common
[
https://issues.apache.org/jira/browse/HADOOP-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055115#comment-14055115
]
Alejandro Abdelnur commented on HADOOP-10771:
-
I'll create 2 subtasks, one to get the DT stuff in common, another for httpfs
to use the one from common.
> Refactor HTTP delegation support out of httpfs to common
>
>
> Key: HADOOP-10771
> URL: https://issues.apache.org/jira/browse/HADOOP-10771
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>
> HttpFS implements delegation token support in {{AuthenticationFilter}} &
> {{AuthenticationHandler}} subclasses.
> For HADOOP-10770 we need similar functionality for KMS.
> Not to duplicate code, we should refactor existing code to common.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-10507:
---
Hadoop Flags: Reviewed
+1 for the patch. After Jenkins runs the latest version, I'll commit it and
co-credit the two of you. Thanks!
> FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
> specified
> --
>
> Key: HADOOP-10507
> URL: https://issues.apache.org/jira/browse/HADOOP-10507
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Stephen Chu
>Assignee: sathish
>Priority: Minor
> Attachments: HADOOP-10507.002.patch, HDFS-6205-0001.patch,
> HDFS-6205.patch
>
>
> If users don't specify the perm of an acl when using the FsShell's setfacl
> command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
> {code}
> [root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
> -setfacl: Fatal internal error
> java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
> at
> org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
> at
> org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
> [root@hdfs-nfs ~]#
> {code}
> An improvement would be if it returned something like this:
> {code}
> [root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
> -setfacl: Invalid permission in : user:bob:rww
> Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x }
> ]|[--set ]
> [root@hdfs-nfs ~]#
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055098#comment-14055098 ] Alejandro Abdelnur commented on HADOOP-10720: - *CommonConfigurationKeysPublic.java:* * there is a KMS constant in KMSClientProvider, with the prefix constant defined there. we should bring that here (may be a simple JIRA to be done before this one). * Javadocs typo 'Defalt' should be 'Default'. * Default cache size should be a bit higher, I’d say 500. * Indicate the expected time unit of the keys in the javadocs and the state the default value with the closest unit, i.e 4320 is 12hrs. * Have a preceding comment > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.1.patch, HADOOP-10720.2.patch, > HADOOP-10720.3.patch, HADOOP-10720.4.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Chu updated HADOOP-10507:
-
Attachment: HADOOP-10507.002.patch
Updated Sathish's patch to fix indenting. I kept the runCommand assertion
consistent with the rest of the test though (4 spaces for next line instead of
2). LMK if it should technically be 2.
[~cnauroth], it's been a while since you reviewed, but I think this patch is
still valid. Compiled successfully and ran TestAclCommands successfully.
> FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
> specified
> --
>
> Key: HADOOP-10507
> URL: https://issues.apache.org/jira/browse/HADOOP-10507
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Stephen Chu
>Assignee: sathish
>Priority: Minor
> Attachments: HADOOP-10507.002.patch, HDFS-6205-0001.patch,
> HDFS-6205.patch
>
>
> If users don't specify the perm of an acl when using the FsShell's setfacl
> command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
> {code}
> [root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
> -setfacl: Fatal internal error
> java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
> at
> org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
> at
> org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
> [root@hdfs-nfs ~]#
> {code}
> An improvement would be if it returned something like this:
> {code}
> [root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
> -setfacl: Invalid permission in : user:bob:rww
> Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x }
> ]|[--set ]
> [root@hdfs-nfs ~]#
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10793) Key Shell args use double dash style
[ https://issues.apache.org/jira/browse/HADOOP-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055090#comment-14055090 ] Alejandro Abdelnur commented on HADOOP-10793: - [~lmccay], I think we should be consistent with the rest of hadoop shell tools, both KP and CP are still in trunk, so it is easy to align them without breaking compatibility. > Key Shell args use double dash style > > > Key: HADOOP-10793 > URL: https://issues.apache.org/jira/browse/HADOOP-10793 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder > > Follow-on from HADOOP-10736 as per [~andrew.wang] - the key shell uses the > gnu double dash style for command line args, while other command line > programs use a single dash. Consider changing this, and consider another > argument parsing scheme, like the CommandLine class. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10793) Key Shell args use double dash style
[ https://issues.apache.org/jira/browse/HADOOP-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055084#comment-14055084 ] Larry McCay commented on HADOOP-10793: -- Hi [~yoderme] - the double dash style was intentionally used for this shell and is aligned with the CredentialShell for the Credential Provider API. I would suggest adding single dash arguments for shortened names if you would rather that style single dashes. I don't believe that we should be limited to the styles used in previous tooling when another is preferred and popular. In addition, what is the value of replacing the command line parsing mechanism? This was modeled after an existing tool and serves the purpose well - afaict? > Key Shell args use double dash style > > > Key: HADOOP-10793 > URL: https://issues.apache.org/jira/browse/HADOOP-10793 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder > > Follow-on from HADOOP-10736 as per [~andrew.wang] - the key shell uses the > gnu double dash style for command line args, while other command line > programs use a single dash. Consider changing this, and consider another > argument parsing scheme, like the CommandLine class. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10798) globStatus() does not return sorted list of files
[
https://issues.apache.org/jira/browse/HADOOP-10798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055082#comment-14055082
]
Felix Borchers commented on HADOOP-10798:
-
Tested on local filesystem. works on a macosx, but not on linux.
> globStatus() does not return sorted list of files
> -
>
> Key: HADOOP-10798
> URL: https://issues.apache.org/jira/browse/HADOOP-10798
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Felix Borchers
>Priority: Minor
>
> (FileSystem) globStatus() does not return a sorted file list anymore.
> But the API says: " ... Results are sorted by their names."
> Seems to be lost, when the Globber Object was introduced. Can't find a sort
> in actual code.
> code to check this behavior:
> {code}
> Configuration conf = new Configuration();
> FileSystem fs = FileSystem.get(conf);
> Path path = new Path("/tmp/" + System.currentTimeMillis());
> fs.mkdirs(path);
> fs.deleteOnExit(path);
> fs.createNewFile(new Path(path, "2"));
> fs.createNewFile(new Path(path, "3"));
> fs.createNewFile(new Path(path, "1"));
> FileStatus[] status = fs.globStatus(new Path(path, "*"));
> Collection list = new ArrayList();
> for (FileStatus f: status) {
> list.add(f.getPath().toString());
> //System.out.println(f.getPath().toString());
> }
> boolean sorted = Ordering.natural().isOrdered(list);
> Assert.assertTrue(sorted);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10782) Typo in DataChecksum classs
[ https://issues.apache.org/jira/browse/HADOOP-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055015#comment-14055015 ] Hudson commented on HADOOP-10782: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1825 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1825/]) HADOOP-10782. Fix typo in DataChecksum class. Contributed by Jingguo Yao. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1608539) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java > Typo in DataChecksum classs > --- > > Key: HADOOP-10782 > URL: https://issues.apache.org/jira/browse/HADOOP-10782 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jingguo Yao >Assignee: Jingguo Yao >Priority: Trivial > Fix For: 2.5.0 > > Attachments: HADOOP-10782.patch > > Original Estimate: 5m > Remaining Estimate: 5m > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10782) Typo in DataChecksum classs
[ https://issues.apache.org/jira/browse/HADOOP-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054955#comment-14054955 ] Hudson commented on HADOOP-10782: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1798 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1798/]) HADOOP-10782. Fix typo in DataChecksum class. Contributed by Jingguo Yao. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1608539) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java > Typo in DataChecksum classs > --- > > Key: HADOOP-10782 > URL: https://issues.apache.org/jira/browse/HADOOP-10782 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jingguo Yao >Assignee: Jingguo Yao >Priority: Trivial > Fix For: 2.5.0 > > Attachments: HADOOP-10782.patch > > Original Estimate: 5m > Remaining Estimate: 5m > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10798) globStatus() does not return sorted list of files
[
https://issues.apache.org/jira/browse/HADOOP-10798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054930#comment-14054930
]
Daryn Sharp commented on HADOOP-10798:
--
Which filesystem? Ie. hdfs or local? GlobStatus should return a sorted list
because it calls listStatus which returns a sorted list. If the glob results
are not sorted, then listStatus is likely the culprit.
> globStatus() does not return sorted list of files
> -
>
> Key: HADOOP-10798
> URL: https://issues.apache.org/jira/browse/HADOOP-10798
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Felix Borchers
>Priority: Minor
>
> (FileSystem) globStatus() does not return a sorted file list anymore.
> But the API says: " ... Results are sorted by their names."
> Seems to be lost, when the Globber Object was introduced. Can't find a sort
> in actual code.
> code to check this behavior:
> {code}
> Configuration conf = new Configuration();
> FileSystem fs = FileSystem.get(conf);
> Path path = new Path("/tmp/" + System.currentTimeMillis());
> fs.mkdirs(path);
> fs.deleteOnExit(path);
> fs.createNewFile(new Path(path, "2"));
> fs.createNewFile(new Path(path, "3"));
> fs.createNewFile(new Path(path, "1"));
> FileStatus[] status = fs.globStatus(new Path(path, "*"));
> Collection list = new ArrayList();
> for (FileStatus f: status) {
> list.add(f.getPath().toString());
> //System.out.println(f.getPath().toString());
> }
> boolean sorted = Ordering.natural().isOrdered(list);
> Assert.assertTrue(sorted);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10798) globStatus() does not return sorted list of files
[
https://issues.apache.org/jira/browse/HADOOP-10798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Borchers updated HADOOP-10798:
Description:
(FileSystem) globStatus() does not return a sorted file list anymore.
But the API says: " ... Results are sorted by their names."
Seems to be lost, when the Globber Object was introduced. Can't find a sort in
actual code.
code to check this behavior:
{code}
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path("/tmp/" + System.currentTimeMillis());
fs.mkdirs(path);
fs.deleteOnExit(path);
fs.createNewFile(new Path(path, "2"));
fs.createNewFile(new Path(path, "3"));
fs.createNewFile(new Path(path, "1"));
FileStatus[] status = fs.globStatus(new Path(path, "*"));
Collection list = new ArrayList();
for (FileStatus f: status) {
list.add(f.getPath().toString());
//System.out.println(f.getPath().toString());
}
boolean sorted = Ordering.natural().isOrdered(list);
Assert.assertTrue(sorted);
{code}
was:
(FileSystem) globStatus() does not return a sorted file list anymore.
But the API says: " ... Results are sorted by their names."
Seems to be lost, when the Globber Object was introduced.
code to check this behavior:
{code}
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path("/tmp/" + System.currentTimeMillis());
fs.mkdirs(path);
fs.deleteOnExit(path);
fs.createNewFile(new Path(path, "2"));
fs.createNewFile(new Path(path, "3"));
fs.createNewFile(new Path(path, "1"));
FileStatus[] status = fs.globStatus(new Path(path, "*"));
Collection list = new ArrayList();
for (FileStatus f: status) {
list.add(f.getPath().toString());
//System.out.println(f.getPath().toString());
}
boolean sorted = Ordering.natural().isOrdered(list);
Assert.assertTrue(sorted);
{code}
> globStatus() does not return sorted list of files
> -
>
> Key: HADOOP-10798
> URL: https://issues.apache.org/jira/browse/HADOOP-10798
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Felix Borchers
>Priority: Minor
>
> (FileSystem) globStatus() does not return a sorted file list anymore.
> But the API says: " ... Results are sorted by their names."
> Seems to be lost, when the Globber Object was introduced. Can't find a sort
> in actual code.
> code to check this behavior:
> {code}
> Configuration conf = new Configuration();
> FileSystem fs = FileSystem.get(conf);
> Path path = new Path("/tmp/" + System.currentTimeMillis());
> fs.mkdirs(path);
> fs.deleteOnExit(path);
> fs.createNewFile(new Path(path, "2"));
> fs.createNewFile(new Path(path, "3"));
> fs.createNewFile(new Path(path, "1"));
> FileStatus[] status = fs.globStatus(new Path(path, "*"));
> Collection list = new ArrayList();
> for (FileStatus f: status) {
> list.add(f.getPath().toString());
> //System.out.println(f.getPath().toString());
> }
> boolean sorted = Ordering.natural().isOrdered(list);
> Assert.assertTrue(sorted);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10798) globStatus() does not return sorted list of files
Felix Borchers created HADOOP-10798:
---
Summary: globStatus() does not return sorted list of files
Key: HADOOP-10798
URL: https://issues.apache.org/jira/browse/HADOOP-10798
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Felix Borchers
Priority: Minor
(FileSystem) globStatus() does not return a sorted file list anymore.
But the API says: " ... Results are sorted by their names."
Seems to be lost, when the Globber Object was introduced.
code to check this behavior:
{code}
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path("/tmp/" + System.currentTimeMillis());
fs.mkdirs(path);
fs.deleteOnExit(path);
fs.createNewFile(new Path(path, "2"));
fs.createNewFile(new Path(path, "3"));
fs.createNewFile(new Path(path, "1"));
FileStatus[] status = fs.globStatus(new Path(path, "*"));
Collection list = new ArrayList();
for (FileStatus f: status) {
list.add(f.getPath().toString());
//System.out.println(f.getPath().toString());
}
boolean sorted = Ordering.natural().isOrdered(list);
Assert.assertTrue(sorted);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-1540) distcp should support an exclude list
[ https://issues.apache.org/jira/browse/HADOOP-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054919#comment-14054919 ] Laurent EDEL commented on HADOOP-1540: -- Event if this should have been implemented in https://issues.apache.org/jira/browse/MAPREDUCE-5014, it's apparently not. This would be very useful, for example not trying to copy Flume files that are not flushed yet (i.e. exclude .tmp files) > distcp should support an exclude list > - > > Key: HADOOP-1540 > URL: https://issues.apache.org/jira/browse/HADOOP-1540 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Senthil Subramanian >Priority: Minor > > There should be a way to ignore specific paths (eg: those that have already > been copied over under the current srcPath). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Sivachenko updated HADOOP-10797: --- Status: Patch Available (was: Open) I am attaching a patch (against trunk) > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10797) Hardcoded path to "bash" is not portable
[ https://issues.apache.org/jira/browse/HADOOP-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Sivachenko updated HADOOP-10797: --- Attachment: bash.patch > Hardcoded path to "bash" is not portable > > > Key: HADOOP-10797 > URL: https://issues.apache.org/jira/browse/HADOOP-10797 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: bash.patch > > > Most of shell scripts use shebang ling in the following format: > #!/usr/bin/env bash > But some scripts contain hardcoded "/bin/bash" which is not portable. > Please use #!/usr/bin/env bash instead for portability. > PS: it would be much better to switch to standard Bourne Shell /bin/sh, do > these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10797) Hardcoded path to "bash" is not portable
Dmitry Sivachenko created HADOOP-10797: -- Summary: Hardcoded path to "bash" is not portable Key: HADOOP-10797 URL: https://issues.apache.org/jira/browse/HADOOP-10797 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Reporter: Dmitry Sivachenko Most of shell scripts use shebang ling in the following format: #!/usr/bin/env bash But some scripts contain hardcoded "/bin/bash" which is not portable. Please use #!/usr/bin/env bash instead for portability. PS: it would be much better to switch to standard Bourne Shell /bin/sh, do these scripts really need bash? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10782) Typo in DataChecksum classs
[ https://issues.apache.org/jira/browse/HADOOP-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054846#comment-14054846 ] Hudson commented on HADOOP-10782: - SUCCESS: Integrated in Hadoop-Yarn-trunk #607 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/607/]) HADOOP-10782. Fix typo in DataChecksum class. Contributed by Jingguo Yao. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1608539) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java > Typo in DataChecksum classs > --- > > Key: HADOOP-10782 > URL: https://issues.apache.org/jira/browse/HADOOP-10782 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jingguo Yao >Assignee: Jingguo Yao >Priority: Trivial > Fix For: 2.5.0 > > Attachments: HADOOP-10782.patch > > Original Estimate: 5m > Remaining Estimate: 5m > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Sivachenko updated HADOOP-10781: --- Attachment: getgrouplist.patch I am attaching a patch in standard format for review. > Unportable getgrouplist() usage breaks FreeBSD > -- > > Key: HADOOP-10781 > URL: https://issues.apache.org/jira/browse/HADOOP-10781 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > Attachments: getgrouplist.patch > > > getgrouplist() has different return values on Linux and FreeBSD: > Linux: either the number of groups (positive) or -1 on error > FreeBSD: 0 on success or -1 on error > The return value of getgrouplist() is analyzed in Linux-specific way in > hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, > in function hadoop_user_info_getgroups() which breaks FreeBSD. > In this function you have 3 choices for the return value > ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, > uinfo->gids, &ngroups); > 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change > this to ret >= 0 > 2) First condition is false and ret != -1: impossible according to manpage > 3) ret == 1 -- OK for both Linux and FreeBSD > So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd > case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10780) namenode throws java.lang.OutOfMemoryError upon DatanodeProtocol.versionRequest from datanode
[ https://issues.apache.org/jira/browse/HADOOP-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Sivachenko updated HADOOP-10780: --- Attachment: buf_sz.patch Here is a patch in standard format for review. > namenode throws java.lang.OutOfMemoryError upon > DatanodeProtocol.versionRequest from datanode > - > > Key: HADOOP-10780 > URL: https://issues.apache.org/jira/browse/HADOOP-10780 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.1 > Environment: FreeBSD-10/stable > openjdk version "1.7.0_60" > OpenJDK Runtime Environment (build 1.7.0_60-b19) > OpenJDK 64-Bit Server VM (build 24.60-b09, mixed mode) >Reporter: Dmitry Sivachenko > Attachments: buf_sz.patch > > > I am trying hadoop-2.4.1 on FreeBSD-10/stable. > namenode starts up, but after first datanode contacts it, it throws an > exception. > All limits seem to be high enough: > % limits -a > Resource limits (current): > cputime infinity secs > filesize infinity kB > datasize 33554432 kB > stacksize 524288 kB > coredumpsize infinity kB > memoryuseinfinity kB > memorylocked infinity kB > maxprocesses 122778 > openfiles 14 > sbsize infinity bytes > vmemoryuse infinity kB > pseudo-terminals infinity > swapuse infinity kB > 14944 1 S0:06.59 /usr/local/openjdk7/bin/java -Dproc_namenode > -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop > -Dhadoop.log.file=hadoop-hdfs-namenode-nezabudka3-00.log > -Dhadoop.home.dir=/usr/local -Dhadoop.id.str=hdfs > -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml > -Djava.net.preferIPv4Stack=true -Xmx32768m -Xms32768m > -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m > -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m > -Djava.library.path=/usr/local/lib -Dhadoop.security.logger=INFO,RFAS > org.apache.hadoop.hdfs.server.namenode.NameNode > From the namenode's log: > 2014-07-03 23:28:15,070 WARN [IPC Server handler 5 on 8020] ipc.Server > (Server.java:run(2032)) - IPC Server handler 5 on 8020, call > org.apache.hadoop.hdfs.server.protocol.Datano > deProtocol.versionRequest from 5.255.231.209:57749 Call#842 Retry#0 > java.lang.OutOfMemoryError > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Native > Method) > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(JniBasedUnixGroupsMapping.java:80) > at > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50) > at org.apache.hadoop.security.Groups.getGroups(Groups.java:139) > at > org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1417) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:81) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3331) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkSuperuserPrivilege(FSNamesystem.java:5491) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.versionRequest(NameNodeRpcServer.java:1082) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.versionRequest(DatanodeProtocolServerSideTranslatorPB.java:234) > at > org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28069) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > I did not have such an issue with hadoop-1.2.1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10796) Porting Hadoop to FreeBSD
Dmitry Sivachenko created HADOOP-10796: -- Summary: Porting Hadoop to FreeBSD Key: HADOOP-10796 URL: https://issues.apache.org/jira/browse/HADOOP-10796 Project: Hadoop Common Issue Type: Bug Environment: FreeBSD Reporter: Dmitry Sivachenko This is "high"-level issue to accumulate other issues about porting Hadoop to FreeBSD in one place. As suggested by Steve Loughran. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10795) unale to build hadoop 2.4.1(redhat5.8 x64)
moses.wang created HADOOP-10795: --- Summary: unale to build hadoop 2.4.1(redhat5.8 x64) Key: HADOOP-10795 URL: https://issues.apache.org/jira/browse/HADOOP-10795 Project: Hadoop Common Issue Type: Bug Components: build Environment: OS version rehat 5.8 x64 maven version 3.3.1 java version jdk 1.7_15 for x64 Reporter: moses.wang unale to build hadoop 2.4.1(redhat5.8 x64) WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project:pom:2.4.1 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ line 1015, column 15 [WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-common:jar:2.4.1 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-surefire-plugin @ line 479, column 15 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ org.apache.hadoop:hadoop-project:2.4.1, /home/software/Server/hadoop-2.4.1-src/hadoop-project/pom.xml, line 1015, column 15 WARNING] /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/FastByteComparisons.java:[25,15] Unsafe is internal proprietary API and may be removed in a future release [WARNING] /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[42,15] Unsafe is internal proprietary API and may be removed in a future release [WARNING] /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java:[21,15] Signal is internal proprietary API and may be removed in a future release [WARNING] /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java:[22,15] SignalHandler is internal proprietary API and may be removed in a future release [WARNING] /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:[22,24] AlgorithmId is internal proprietary API and may be removed in a future release [WARNING] /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:[23,24] CertificateAlgorithmId is internal proprietary API and may be removed in a future release testROBufferDirAndRWBufferDir[1](org.apache.hadoop.fs.TestLocalDirAllocator) Time elapsed: 0.014 sec <<< FAILURE! [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-common: There are test failures. [ERROR] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10794) A hadoop cluster needs clock synchronization
[
https://issues.apache.org/jira/browse/HADOOP-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054657#comment-14054657
]
Steve Loughran commented on HADOOP-10794:
-
I'm not sure if we can/should be doing much more than recommend NTP and setting
time zones properly (if a host has it's TZ env var wrong then it can appear to
be hours out, even if it is in sync). NTP sets it host-wide, calculates local
clock drift and compensates for it, and has good availability.
FWIW Ant's diagnostics method not only prints out TZ info, it looks at the temp
dir and compares its clock with the local time, as the situation "NAS filestore
in different time to host" can cause chaos with dependency logic
{code}
---
Temp dir
---
Temp dir is /var/folders/57/xyts0qt105z1f1k0twk6rd8mgq/T/
Temp dir is writeable
Temp dir alignment with system clock is -819 ms
---
Locale information
---
Timezone Greenwich Mean Time offset=360
{code}
For Hadoop, the problem will be worst in VMs, as their clocks will be jerky and
may even go backwards if a VM is moved to another physical host. NTP isn't so
good there as its a a use case it doesn't expect.
Ignoring that, the real risk is "ops think a machine is syncing its clock with
NTP but isn't". I've seen this happen.
We may want hadoop to help catch that by looking at clocks across a cluster and
warning if some is "significantly" off. This could be done with the RM catching
the times of hosts when they report in -and flagging when one is beyond a
configured threshold. Then it'll be left to that ops team to fix it however
they do it in the cluster.
> A hadoop cluster needs clock synchronization
>
>
> Key: HADOOP-10794
> URL: https://issues.apache.org/jira/browse/HADOOP-10794
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Zhijie Shen
>
> As a distributed system, a hadoop cluster wants the clock on all the
> participating hosts synchronized. Otherwise, some problems might happen. For
> example, in YARN-2251, due to the clock on the host for the task container
> falls behind that on the host of the AM container, the computed elapsed time
> (the diff between the timestamps produced on two hosts) becomes negative.
> In YARN-2251, we tried to mask the negative elapsed time. However, we should
> seek for a decent long term solution, such as providing mechanism to do and
> check clock synchronization.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10792) Add FileSystem#closeIfNotReferred method
[ https://issues.apache.org/jira/browse/HADOOP-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054642#comment-14054642 ] Steve Loughran commented on HADOOP-10792: - I feel your pain, as I've hit it too. You can ask for a non-shared copy of a FileSystem instance by setting the property "fs.%s.impl.disable.cache" for the config you use when you ask for the FS. I recommend you do that in libraries, or rely on the JVM shutdown logic to trigger cleanup. > Add FileSystem#closeIfNotReferred method > > > Key: HADOOP-10792 > URL: https://issues.apache.org/jira/browse/HADOOP-10792 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kousuke Saruta > > FileSystem#close closes FileSystem even if the same instance of FileSystem is > referred by someone. > For instance, a library using FileSystem calls FileSystem.get, and a program > using the library calls FileSystem.get, both of instances of FileSystem is > same. > When the library and the program is implemented as different threads and one > calls FileSystem.close, another fails most of operations of FileSystem. > So, we need the method like cloesIfNotReferred, which closes FileSystem only > if a instance of FileSystem is not referred by anyone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10794) A hadoop cluster needs clock synchronization
[ https://issues.apache.org/jira/browse/HADOOP-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054631#comment-14054631 ] Liang Xie commented on HADOOP-10794: seems running NTP service is a best practice here? > A hadoop cluster needs clock synchronization > > > Key: HADOOP-10794 > URL: https://issues.apache.org/jira/browse/HADOOP-10794 > Project: Hadoop Common > Issue Type: Bug >Reporter: Zhijie Shen > > As a distributed system, a hadoop cluster wants the clock on all the > participating hosts synchronized. Otherwise, some problems might happen. For > example, in YARN-2251, due to the clock on the host for the task container > falls behind that on the host of the AM container, the computed elapsed time > (the diff between the timestamps produced on two hosts) becomes negative. > In YARN-2251, we tried to mask the negative elapsed time. However, we should > seek for a decent long term solution, such as providing mechanism to do and > check clock synchronization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10730) Login with kerberos should always set refreshKrb5Config to true
[ https://issues.apache.org/jira/browse/HADOOP-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ycbai updated HADOOP-10730: --- Priority: Critical (was: Major) > Login with kerberos should always set refreshKrb5Config to true > --- > > Key: HADOOP-10730 > URL: https://issues.apache.org/jira/browse/HADOOP-10730 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: ycbai >Priority: Critical > > When logined with kerberos by "kinit" way, it never refreshes the kerberos > configurations. It is really often needed to be refreshed same as "keytab" > way(The bug of "keytab" has been fixed by HADOOP-6947). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10736) Add key attributes to the key shell
[
https://issues.apache.org/jira/browse/HADOOP-10736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054626#comment-14054626
]
Hadoop QA commented on HADOOP-10736:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654476/HADOOP-10736.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverControllerStress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4223//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4223//console
This message is automatically generated.
> Add key attributes to the key shell
> ---
>
> Key: HADOOP-10736
> URL: https://issues.apache.org/jira/browse/HADOOP-10736
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Mike Yoder
>Assignee: Mike Yoder
> Fix For: 3.0.0
>
> Attachments: HADOOP-10736.003.patch, HADOOP-10736.patch,
> HADOOP-10736.patch
>
>
> The recent work in HADOOP-10696 added attribute-value pairs to keys in the
> KMS. Now the key shell needs to be updated to set/get these attributes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10794) A hadoop cluster needs clock synchronization
[ https://issues.apache.org/jira/browse/HADOOP-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated HADOOP-10794: - Description: As a distributed system, a hadoop cluster wants the clock on all the participating hosts synchronized. Otherwise, some problems might happen. For example, in YARN-2251, due to the clock on the host for the task container falls behind that on the host of the AM container, the computed elapsed time (the diff between the timestamps produced on two hosts) becomes negative. In YARN-2251, we tried to mask the negative elapsed time. However, we should seek for a decent long term solution, such as providing mechanism to do and check clock synchronization. was: As a distributed system, a hadoop cluster wants the clock on all the participating hosts synchronized. Otherwise, some problems might happen. For example, in MAPREDUCE-5940, due to the clock on the host for the task container falls behind that on the host of the AM container, the computed elapsed time (the diff between the timestamps produced on two hosts) becomes negative. In MAPREDUCE-5940, we tried to mask the negative elapsed time. However, we should seek for a decent long term solution, such as providing mechanism to do and check clock synchronization. > A hadoop cluster needs clock synchronization > > > Key: HADOOP-10794 > URL: https://issues.apache.org/jira/browse/HADOOP-10794 > Project: Hadoop Common > Issue Type: Bug >Reporter: Zhijie Shen > > As a distributed system, a hadoop cluster wants the clock on all the > participating hosts synchronized. Otherwise, some problems might happen. For > example, in YARN-2251, due to the clock on the host for the task container > falls behind that on the host of the AM container, the computed elapsed time > (the diff between the timestamps produced on two hosts) becomes negative. > In YARN-2251, we tried to mask the negative elapsed time. However, we should > seek for a decent long term solution, such as providing mechanism to do and > check clock synchronization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10794) A hadoop cluster needs clock synchronization
Zhijie Shen created HADOOP-10794: Summary: A hadoop cluster needs clock synchronization Key: HADOOP-10794 URL: https://issues.apache.org/jira/browse/HADOOP-10794 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen As a distributed system, a hadoop cluster wants the clock on all the participating hosts synchronized. Otherwise, some problems might happen. For example, in MAPREDUCE-5940, due to the clock on the host for the task container falls behind that on the host of the AM container, the computed elapsed time (the diff between the timestamps produced on two hosts) becomes negative. In MAPREDUCE-5940, we tried to mask the negative elapsed time. However, we should seek for a decent long term solution, such as providing mechanism to do and check clock synchronization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10793) Key Shell args use double dash style
Mike Yoder created HADOOP-10793: --- Summary: Key Shell args use double dash style Key: HADOOP-10793 URL: https://issues.apache.org/jira/browse/HADOOP-10793 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Mike Yoder Follow-on from HADOOP-10736 as per [~andrew.wang] - the key shell uses the gnu double dash style for command line args, while other command line programs use a single dash. Consider changing this, and consider another argument parsing scheme, like the CommandLine class. -- This message was sent by Atlassian JIRA (v6.2#6252)
