[jira] [Assigned] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu reassigned HADOOP-13441:
---
Assignee: Yuanbo Liu
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Yuanbo Liu
>Priority: Minor
> Labels: documentation
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
> {{hadoop.security.credstore.java-keystore-provider.password-file}}
> This is the password file to protect credential files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400383#comment-15400383
]
Yuanbo Liu commented on HADOOP-13441:
-
[~jojochuang]Thank you for filing this, I'd like to work on this issue if you
don't mind.
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Priority: Minor
> Labels: documentation
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
> {{hadoop.security.credstore.java-keystore-provider.password-file}}
> This is the password file to protect credential files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13439) Fix race between TestMetricsSystemImpl and TestGangliaMetrics
[
https://issues.apache.org/jira/browse/HADOOP-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400285#comment-15400285
]
Chen Liang edited comment on HADOOP-13439 at 7/30/16 12:11 AM:
---
Hi [~iwasakims],
Any updates on this issue? As I found this JIRA and HADOOP-12588 interesting,
do you remind I assign this to myself and work on this?
was (Author: vagarychen):
Hi [~iwasakims],
Any updates on this issue? As I found this JIRA and HADOOP-12588 interesting,
do you remind I assign this to myself and work on this?
- Chen
> Fix race between TestMetricsSystemImpl and TestGangliaMetrics
> -
>
> Key: HADOOP-13439
> URL: https://issues.apache.org/jira/browse/HADOOP-13439
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Priority: Minor
>
> TestGangliaMetrics#testGangliaMetrics2 set *.period to 120 but 8 was used.
> {noformat}
> 2016-06-27 15:21:31,480 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:startTimer(375)) - Scheduled snapshot period at 8
> second(s).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13439) Fix race between TestMetricsSystemImpl and TestGangliaMetrics
[
https://issues.apache.org/jira/browse/HADOOP-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400285#comment-15400285
]
Chen Liang commented on HADOOP-13439:
-
Hi [~iwasakims],
Any updates on this issue? As I found this JIRA and HADOOP-12588 interesting,
do you remind I assign this to myself and work on this?
- Chen
> Fix race between TestMetricsSystemImpl and TestGangliaMetrics
> -
>
> Key: HADOOP-13439
> URL: https://issues.apache.org/jira/browse/HADOOP-13439
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Priority: Minor
>
> TestGangliaMetrics#testGangliaMetrics2 set *.period to 120 but 8 was used.
> {noformat}
> 2016-06-27 15:21:31,480 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:startTimer(375)) - Scheduled snapshot period at 8
> second(s).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13443) KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension
[
https://issues.apache.org/jira/browse/HADOOP-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400269#comment-15400269
]
Xiao Chen commented on HADOOP-13443:
Thanks very much for finding and fixing this,
[~anthony.young-gar...@cloudera.com]!
Patch looks good in general. A few comments:
- Before this patch, if the {{KeyProvider}} itself implements
{{CryptoExtension}}, it will be used, no matter whether it implements
{{KeyProviderExtension}} or not. This behavior is changed from this patch, is
it what we intend to do? IMHO we should check on {{CryptoExtension}} first, and
add the {{KeyProviderExtension}} check as a fall back of the former.
- In patch 1, if {{keyProvider instanceof KeyProviderExtension == true}} but
{{keyProviderExtension.getKeyProvider() instanceof
KeyProviderCryptoExtension.CryptoExtension == false}}, {{cryptoExtension}} will
end up being {{null}}. Let's make sure the default is used in any case.
- Please fix the checkstyle warnings.
> KMS uses DefaultCryptoProvider when active keyprovider implements
> KeyProviderCryptoExtension
>
>
> Key: HADOOP-13443
> URL: https://issues.apache.org/jira/browse/HADOOP-13443
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Anthony Young-Garner
>Assignee: Anthony Young-Garner
>Priority: Minor
> Attachments: HADOOP-13443.patch
>
>
> By default, the KMS wraps the active key provider in a CachingKeyProvider at
> runtime. This prevents the
> KeyProviderCryptoExtension.createKeyProviderCryptoExtension method from ever
> detecting whether the active key provider implements
> theKeyProviderCryptoExtension interface. Therefore, the
> DefaultCryptoExtension is always used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy
[
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400217#comment-15400217
]
Mingliang Liu edited comment on HADOOP-13436 at 7/29/16 11:25 PM:
--
I'm in favor of making {{RetryPolicy}} an abstract class. In Java we can't add
those methods to an interface. Implementing final
{{hashCode}}/{{equals}}/{{toString}} methods; or make {{hashCode}}/{{equals}}
abstract, both approach should work IMO.
How about different implementations in {{branch-2}} and {{trunk}}? In
{{trunk}}, we make {{RetryPolicy}} an abstract class, while in {{branch-2}}, we
add a new {{AbstractRetryPolicy}} class and revisit all the policies by
extending for correctness. This way, we can defend the compatibility vs. fix
debate for {{branch-2}} but still, make the future a better place.
was (Author: liuml07):
I'm in favor of making {{RetryPolicy}} an abstract class. In Java we can't add
those methods to an interface. Implementing final
{{hashCode}}/{{equals}}/{{toString}} methods; or make {{hashCode}}/{{equals}}
abstract, both approach should work IMO.
How about different implementations in {{branch-2}} and {{trunk}}? In
{{branch-2}}, we make {{RetryPolicy}} an abstract class, while in {{branch-2}},
we add a new {{AbstractRetryPolicy}} class and revisit all the policies by
extending for correctness. This way, we can defend the compatibility vs. fix
debate for {{branch-2}} but still, make the future a better place.
> RPC connections are leaking due to missing equals override in
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat}
> enabled. Internally, Client#getConnection is doing lookup relying on
> ConnectionId#equals, which composes checking Subclass-of-RetryPolicy#equals.
> If subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every
> instances of RetryPolicy with equivalent fields' values (e.g.
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy is called:
> {noformat}
> at
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy
[
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400217#comment-15400217
]
Mingliang Liu commented on HADOOP-13436:
I'm in favor of making {{RetryPolicy}} an abstract class. In Java we can't add
those methods to an interface. Implementing final
{{hashCode}}/{{equals}}/{{toString}} methods; or make {{hashCode}}/{{equals}}
abstract, both approach should work IMO.
How about different implementations in {{branch-2}} and {{trunk}}? In
{{branch-2}}, we make {{RetryPolicy}} an abstract class, while in {{branch-2}},
we add a new {{AbstractRetryPolicy}} class and revisit all the policies by
extending for correctness. This way, we can defend the compatibility vs. fix
debate for {{branch-2}} but still, make the future a better place.
> RPC connections are leaking due to missing equals override in
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat}
> enabled. Internally, Client#getConnection is doing lookup relying on
> ConnectionId#equals, which composes checking Subclass-of-RetryPolicy#equals.
> If subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every
> instances of RetryPolicy with equivalent fields' values (e.g.
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy is called:
> {noformat}
> at
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at
>
[jira] [Commented] (HADOOP-13443) KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension
[
https://issues.apache.org/jira/browse/HADOOP-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400170#comment-15400170
]
Hadoop QA commented on HADOOP-13443:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 10 new + 13 unchanged - 0 fixed = 23 total (was 13) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821095/HADOOP-13443.patch |
| JIRA Issue | HADOOP-13443 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 39955467d57e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10126/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10126/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10126/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMS uses DefaultCryptoProvider when active keyprovider implements
> KeyProviderCryptoExtension
>
>
> Key: HADOOP-13443
> URL: https://issues.apache.org/jira/browse/HADOOP-13443
> Project: Hadoop Common
> Issue Type:
[jira] [Commented] (HADOOP-13434) Add quoting to Shell class
[
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400166#comment-15400166
]
Hadoop QA commented on HADOOP-13434:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 48 unchanged - 11 fixed = 48 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 30s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13434 |
| GITHUB PR | https://github.com/apache/hadoop/pull/119 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a2c95e691938 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10125/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10125/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10125/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments:
[jira] [Commented] (HADOOP-13444) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[
https://issues.apache.org/jira/browse/HADOOP-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400162#comment-15400162
]
Hadoop QA commented on HADOOP-13444:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-common-project: The patch generated 18
new + 912 unchanged - 12 fixed = 930 total (was 924) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-minikdc in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 50s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.ipc.TestRPCWaitForProxy |
| | hadoop.security.TestGroupsCaching |
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821088/HADOOP-13444.patch |
| JIRA Issue | HADOOP-13444 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2b5277e328e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-13081) add the ability to create multiple UGIs/subjects from one kerberos login
[
https://issues.apache.org/jira/browse/HADOOP-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400147#comment-15400147
]
Hadoop QA commented on HADOOP-13081:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 54s{color}
| {color:red} root generated 1 new + 709 unchanged - 1 fixed = 710 total (was
710) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 17s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821087/HADOOP-13081.02.patch
|
| JIRA Issue | HADOOP-13081 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 96ca3687bfbc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10123/artifact/patchprocess/diff-compile-javac-root.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10123/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10123/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10123/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10123/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add
[jira] [Updated] (HADOOP-13434) Add quoting to Shell class
[ https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HADOOP-13434: --- Attachment: HADOOP-13434.patch I made some tweaks to make check style happy. > Add quoting to Shell class > -- > > Key: HADOOP-13434 > URL: https://issues.apache.org/jira/browse/HADOOP-13434 > Project: Hadoop Common > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HADOOP-13434.patch, HADOOP-13434.patch, > HADOOP-13434.patch > > > The Shell class makes assumptions that the parameters won't have spaces or > other special characters, even when it invokes bash. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13443) KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension
[ https://issues.apache.org/jira/browse/HADOOP-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Young-Garner updated HADOOP-13443: -- Status: Patch Available (was: Open) > KMS uses DefaultCryptoProvider when active keyprovider implements > KeyProviderCryptoExtension > > > Key: HADOOP-13443 > URL: https://issues.apache.org/jira/browse/HADOOP-13443 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Anthony Young-Garner >Assignee: Anthony Young-Garner >Priority: Minor > Attachments: HADOOP-13443.patch > > > By default, the KMS wraps the active key provider in a CachingKeyProvider at > runtime. This prevents the > KeyProviderCryptoExtension.createKeyProviderCryptoExtension method from ever > detecting whether the active key provider implements > theKeyProviderCryptoExtension interface. Therefore, the > DefaultCryptoExtension is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13443) KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension
[ https://issues.apache.org/jira/browse/HADOOP-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Young-Garner updated HADOOP-13443: -- Attachment: HADOOP-13443.patch > KMS uses DefaultCryptoProvider when active keyprovider implements > KeyProviderCryptoExtension > > > Key: HADOOP-13443 > URL: https://issues.apache.org/jira/browse/HADOOP-13443 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Anthony Young-Garner >Assignee: Anthony Young-Garner >Priority: Minor > Attachments: HADOOP-13443.patch > > > By default, the KMS wraps the active key provider in a CachingKeyProvider at > runtime. This prevents the > KeyProviderCryptoExtension.createKeyProviderCryptoExtension method from ever > detecting whether the active key provider implements > theKeyProviderCryptoExtension interface. Therefore, the > DefaultCryptoExtension is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13081) add the ability to create multiple UGIs/subjects from one kerberos login
[
https://issues.apache.org/jira/browse/HADOOP-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400072#comment-15400072
]
Chris Nauroth commented on HADOOP-13081:
Nice catch on cloning the Hadoop {{Credentials}} instances! Those are mutable.
{code}
Set set = new HashSet<>(old.size());
{code}
I assume this was intended to size the new set identical to the old one to
prevent reallocations while adding elements. Unfortunately, the constructor
argument is the initial capacity, not the expected size. With the default load
factor of 0.75, that means passing the old set's size actually is highly likely
to cause one reallocation while adding elements.
I think using the no-args constructor is acceptable here, given that the sets
are expected to be small. If you feel it's really important to control the
size, then you could use something like Guava's logic:
https://github.com/google/guava/blob/v11.0.2/guava/src/com/google/common/collect/Maps.java#L106-L120
(Please don't call Guava directly here though. We're starting to try to limit
our dependency on it.)
I think this will be ready to go after addressing that and the last round of
pre-commit nitpicks.
> add the ability to create multiple UGIs/subjects from one kerberos login
>
>
> Key: HADOOP-13081
> URL: https://issues.apache.org/jira/browse/HADOOP-13081
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HADOOP-13081.01.patch, HADOOP-13081.02.patch,
> HADOOP-13081.02.patch, HADOOP-13081.patch
>
>
> We have a scenario where we log in with kerberos as a certain user for some
> tasks, but also want to add tokens to the resulting UGI that would be
> specific to each task. We don't want to authenticate with kerberos for every
> task.
> I am not sure how this can be accomplished with the existing UGI interface.
> Perhaps some clone method would be helpful, similar to createProxyUser minus
> the proxy stuff; or it could just relogin anew from ticket cache.
> getUGIFromTicketCache seems like the best option in existing code, but there
> doesn't appear to be a consistent way of handling ticket cache location - the
> above method, that I only see called in test, is using a config setting that
> is not used anywhere else, and the env variable for the location that is used
> in the main ticket cache related methods is not set uniformly on all paths -
> therefore, trying to find the correct ticket cache and passing it via the
> config setting to getUGIFromTicketCache seems even hackier than doing the
> clone via reflection ;) Moreover, getUGIFromTicketCache ignores the user
> parameter on the main path - it logs a warning for multiple principals and
> then logs in with first available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13081) add the ability to create multiple UGIs/subjects from one kerberos login
[ https://issues.apache.org/jira/browse/HADOOP-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HADOOP-13081: -- Attachment: HADOOP-13081.02.patch Updating the patch to remove tabs > add the ability to create multiple UGIs/subjects from one kerberos login > > > Key: HADOOP-13081 > URL: https://issues.apache.org/jira/browse/HADOOP-13081 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HADOOP-13081.01.patch, HADOOP-13081.02.patch, > HADOOP-13081.02.patch, HADOOP-13081.patch > > > We have a scenario where we log in with kerberos as a certain user for some > tasks, but also want to add tokens to the resulting UGI that would be > specific to each task. We don't want to authenticate with kerberos for every > task. > I am not sure how this can be accomplished with the existing UGI interface. > Perhaps some clone method would be helpful, similar to createProxyUser minus > the proxy stuff; or it could just relogin anew from ticket cache. > getUGIFromTicketCache seems like the best option in existing code, but there > doesn't appear to be a consistent way of handling ticket cache location - the > above method, that I only see called in test, is using a config setting that > is not used anywhere else, and the env variable for the location that is used > in the main ticket cache related methods is not set uniformly on all paths - > therefore, trying to find the correct ticket cache and passing it via the > config setting to getUGIFromTicketCache seems even hackier than doing the > clone via reflection ;) Moreover, getUGIFromTicketCache ignores the user > parameter on the main path - it logs a warning for multiple principals and > then logs in with first available. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) support cross-region operations in S3a
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400047#comment-15400047
]
Steven K. Wong commented on HADOOP-13336:
-
S3A already supports cross-region, if your config doesn't specify endpoint and
doesn't enable path-style access. I am able to {{hadoop fs -cp}} a file from an
S3 bucket in one region (us-east-1) to another S3 bucket in a different region
(us-west-1).
> support cross-region operations in S3a
> --
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> S3a now supports different regions, by way of declaring the endpoint —but you
> can't do things like read in one region, write back in another (e.g. a distcp
> backup), because only one region can be specified in a configuration.
> If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt
> s3a://b2.seol , then this would be possible.
> Swift does this with a full filesystem binding/config: endpoints, username,
> etc, in the XML file. Would we need to do that much? It'd be simpler
> initially to use a domain suffix of a URL to set the region of a bucket from
> the domain and have the aws library sort the details out itself, maybe with
> some config options for working with non-AWS infra
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13444) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HADOOP-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HADOOP-13444: -- Attachment: HADOOP-13444.patch > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HADOOP-13444 > URL: https://issues.apache.org/jira/browse/HADOOP-13444 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.4 >Reporter: Vincent Poon >Priority: Minor > Fix For: 2.7.4 > > Attachments: HADOOP-13444.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13444) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HADOOP-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HADOOP-13444: -- Fix Version/s: (was: 2.7.2) 2.7.4 Affects Version/s: (was: 2.7.2) 2.7.4 Status: Patch Available (was: Open) > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HADOOP-13444 > URL: https://issues.apache.org/jira/browse/HADOOP-13444 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.4 >Reporter: Vincent Poon >Priority: Minor > Fix For: 2.7.4 > > Attachments: HADOOP-13444.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13081) add the ability to create multiple UGIs/subjects from one kerberos login
[
https://issues.apache.org/jira/browse/HADOOP-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400042#comment-15400042
]
Hadoop QA commented on HADOOP-13081:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 51s{color}
| {color:red} root generated 1 new + 709 unchanged - 1 fixed = 710 total (was
710) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 6 new + 150 unchanged - 0 fixed = 156 total (was 150) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
45s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821080/HADOOP-13081.02.patch
|
| JIRA Issue | HADOOP-13081 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 176ab1e88879 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10122/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10122/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10122/artifact/patchprocess/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10122/artifact/patchprocess/whitespace-tabs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10122/testReport/ |
| modules | C:
[jira] [Created] (HADOOP-13444) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
Vincent Poon created HADOOP-13444: - Summary: Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets Key: HADOOP-13444 URL: https://issues.apache.org/jira/browse/HADOOP-13444 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.7.2 Reporter: Vincent Poon Priority: Minor Fix For: 2.7.2 org.apache.commons.io.Charsets is deprecated in favor of java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13443) KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension
[ https://issues.apache.org/jira/browse/HADOOP-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Young-Garner updated HADOOP-13443: -- Flags: (was: Patch) > KMS uses DefaultCryptoProvider when active keyprovider implements > KeyProviderCryptoExtension > > > Key: HADOOP-13443 > URL: https://issues.apache.org/jira/browse/HADOOP-13443 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Anthony Young-Garner >Assignee: Anthony Young-Garner >Priority: Minor > > By default, the KMS wraps the active key provider in a CachingKeyProvider at > runtime. This prevents the > KeyProviderCryptoExtension.createKeyProviderCryptoExtension method from ever > detecting whether the active key provider implements > theKeyProviderCryptoExtension interface. Therefore, the > DefaultCryptoExtension is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13443) KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension
Anthony Young-Garner created HADOOP-13443: - Summary: KMS uses DefaultCryptoProvider when active keyprovider implements KeyProviderCryptoExtension Key: HADOOP-13443 URL: https://issues.apache.org/jira/browse/HADOOP-13443 Project: Hadoop Common Issue Type: Bug Components: kms Affects Versions: 2.6.0 Reporter: Anthony Young-Garner Assignee: Anthony Young-Garner Priority: Minor By default, the KMS wraps the active key provider in a CachingKeyProvider at runtime. This prevents the KeyProviderCryptoExtension.createKeyProviderCryptoExtension method from ever detecting whether the active key provider implements theKeyProviderCryptoExtension interface. Therefore, the DefaultCryptoExtension is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400011#comment-15400011 ] Kai Zheng commented on HADOOP-11540: Per Aaron's below comments: bq. I'll be +1 once this is addressed, and per Andrew's direction I will commit this to both trunk and the 3.0 release branch. The latest patch has addressed the issue, avoiding the unnecessary license files. I will commit this in early next week as said above if no other concerns. > Raw Reed-Solomon coder using Intel ISA-L library > > > Key: HADOOP-11540 > URL: https://issues.apache.org/jira/browse/HADOOP-11540 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Kai Zheng > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, > HADOOP-11540-v10.patch, HADOOP-11540-v11.patch, HADOOP-11540-v12.patch, > HADOOP-11540-v13.patch, HADOOP-11540-v2.patch, HADOOP-11540-v4.patch, > HADOOP-11540-v5.patch, HADOOP-11540-v6.patch, HADOOP-11540-v7.patch, > HADOOP-11540-v8.patch, HADOOP-11540-v9.patch, > HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel > ISAL-v1.pdf > > > This is to provide RS codec implementation using Intel ISA-L library for > encoding and decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13081) add the ability to create multiple UGIs/subjects from one kerberos login
[ https://issues.apache.org/jira/browse/HADOOP-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HADOOP-13081: -- Attachment: HADOOP-13081.02.patch Added the test and some additional logic to clone Hadoop credentials, which are apparently reused from the set, rather than adding to the set. > add the ability to create multiple UGIs/subjects from one kerberos login > > > Key: HADOOP-13081 > URL: https://issues.apache.org/jira/browse/HADOOP-13081 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HADOOP-13081.01.patch, HADOOP-13081.02.patch, > HADOOP-13081.patch > > > We have a scenario where we log in with kerberos as a certain user for some > tasks, but also want to add tokens to the resulting UGI that would be > specific to each task. We don't want to authenticate with kerberos for every > task. > I am not sure how this can be accomplished with the existing UGI interface. > Perhaps some clone method would be helpful, similar to createProxyUser minus > the proxy stuff; or it could just relogin anew from ticket cache. > getUGIFromTicketCache seems like the best option in existing code, but there > doesn't appear to be a consistent way of handling ticket cache location - the > above method, that I only see called in test, is using a config setting that > is not used anywhere else, and the env variable for the location that is used > in the main ticket cache related methods is not set uniformly on all paths - > therefore, trying to find the correct ticket cache and passing it via the > config setting to getUGIFromTicketCache seems even hackier than doing the > clone via reflection ;) Moreover, getUGIFromTicketCache ignores the user > parameter on the main path - it logs a warning for multiple principals and > then logs in with first available. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13434) Add quoting to Shell class
[ https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399904#comment-15399904 ] ASF GitHub Bot commented on HADOOP-13434: - Github user arp7 commented on the issue: https://github.com/apache/hadoop/pull/119 +1 lgtm. > Add quoting to Shell class > -- > > Key: HADOOP-13434 > URL: https://issues.apache.org/jira/browse/HADOOP-13434 > Project: Hadoop Common > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HADOOP-13434.patch, HADOOP-13434.patch > > > The Shell class makes assumptions that the parameters won't have spaces or > other special characters, even when it invokes bash. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13208) S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories
[
https://issues.apache.org/jira/browse/HADOOP-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399881#comment-15399881
]
Chris Nauroth commented on HADOOP-13208:
[~fabbri], thank you for your code review. This is near the head of my review
queue, so I'll look soon too. Steve is out for several weeks, so it will take
a while before he can respond to code review feedback.
bq. I feel like we need to break up S3AFileSystem.java some.
Yes, I agree, and I think this is a requirement for S3Guard. I have a
breakdown in mind that I plan to propose as a sub-task of HADOOP-13345.
> S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the
> pseudo-tree of directories
>
>
> Key: HADOOP-13208
> URL: https://issues.apache.org/jira/browse/HADOOP-13208
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13208-branch-2-001.patch,
> HADOOP-13208-branch-2-007.patch, HADOOP-13208-branch-2-008.patch,
> HADOOP-13208-branch-2-009.patch, HADOOP-13208-branch-2-010.patch,
> HADOOP-13208-branch-2-011.patch, HADOOP-13208-branch-2-012.patch,
> HADOOP-13208-branch-2-017.patch, HADOOP-13208-branch-2-018.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> A major cost in split calculation against object stores turns out be listing
> the directory tree itself. That's because against S3, it takes S3A two HEADs
> and two lists to list the content of any directory path (2 HEADs + 1 list for
> getFileStatus(); the next list to query the contents).
> Listing a directory could be improved slightly by combining the final two
> listings. However, a listing of a directory tree will still be
> O(directories). In contrast, a recursive {{listFiles()}} operation should be
> implementable by a bulk listing of all descendant paths; one List operation
> per thousand descendants.
> As the result of this call is an iterator, the ongoing listing can be
> implemented within the iterator itself
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13442) Optimize UGI group lookups
[
https://issues.apache.org/jira/browse/HADOOP-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399880#comment-15399880
]
Hadoop QA commented on HADOOP-13442:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 432 unchanged - 4 fixed = 434 total (was 436) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 58s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821047/HADOOP-13442.patch |
| JIRA Issue | HADOOP-13442 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b6a60f66 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10121/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10121/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10121/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10121/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize UGI group lookups
> --
>
> Key: HADOOP-13442
> URL:
[jira] [Commented] (HADOOP-13434) Add quoting to Shell class
[
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399820#comment-15399820
]
Hadoop QA commented on HADOOP-13434:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
1s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 11 new + 49 unchanged - 10 fixed = 60 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 34s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13434 |
| GITHUB PR | https://github.com/apache/hadoop/pull/119 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8ef61c839f69 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95f2b98 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10120/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10120/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10120/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10120/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
>
[jira] [Updated] (HADOOP-13442) Optimize UGI group lookups
[
https://issues.apache.org/jira/browse/HADOOP-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-13442:
-
Status: Patch Available (was: Open)
Will be much more useful with hdfs counterpart.
> Optimize UGI group lookups
> --
>
> Key: HADOOP-13442
> URL: https://issues.apache.org/jira/browse/HADOOP-13442
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HADOOP-13442.patch
>
>
> {{UGI#getGroups}} and its usage is inefficient. The list is unnecessarily
> converted to multiple collections.
> For _every_ invocation, the {{List}} from the group provider is
> converted into a {{LinkedHashSet}} (to de-dup), back to a
> {{String[]}}. Then callers testing for group membership convert back to a
> {{List}}. This should be done once to reduce allocations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13442) Optimize UGI group lookups
[
https://issues.apache.org/jira/browse/HADOOP-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-13442:
-
Attachment: HADOOP-13442.patch
# Changed group provider to cache the de-dupped list instead of the raw list.
# Added new {{UGI#getGroups}} that returns the aforementioned de-duped list
# Changed {{UGI#getPrimaryGroup}} to call {{UGI#getGroups}} to avoid an array
copy
# Removed unnecessary synchronization of {{UGI#getGroups}} method. Required
minor tweak to {{Groups#getGroups}} to be thread-safe. Already used elsewhere
w/o synch, so this just makes it safe. Reduces contention with cached
token->ugi instances.
> Optimize UGI group lookups
> --
>
> Key: HADOOP-13442
> URL: https://issues.apache.org/jira/browse/HADOOP-13442
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HADOOP-13442.patch
>
>
> {{UGI#getGroups}} and its usage is inefficient. The list is unnecessarily
> converted to multiple collections.
> For _every_ invocation, the {{List}} from the group provider is
> converted into a {{LinkedHashSet}} (to de-dup), back to a
> {{String[]}}. Then callers testing for group membership convert back to a
> {{List}}. This should be done once to reduce allocations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[
https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399461#comment-15399461
]
Wei-Chiu Chuang edited comment on HADOOP-13353 at 7/29/16 6:13 PM:
---
I see that when the IOException is thrown, the log message does not preserve
its stacktrace. Could you also update the fix to include the stacktrace?
Basically, change it to:
{code}
catch (IOException ioe) {
LOG.warn("Exception while trying to get password for alias " + alias + ":
", ioe);
}
{code}
was (Author: jojochuang):
I see that when the IOException is thrown, the log message does not preserve
its stacktrace. Could you also update the fix to include the stacktrace?
Basically, change it to:
{code}
catch (IOException ioe) {
LOG.warn("Exception while trying to password for alias " + alias + ": ",
ioe);
}
{code}
> LdapGroupsMapping getPassward shouldn't return null when IOException throws
> ---
>
> Key: HADOOP-13353
> URL: https://issues.apache.org/jira/browse/HADOOP-13353
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Zhaohao Liang
> Labels: supportability
> Attachments: HADOOP-13353.patch
>
>
> When IOException throws in getPassword(), getPassword() return null String,
> this will cause setConf() throws java.lang.NullPointerException when check
> isEmpty() on null string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13441:
-
Description:
A few properties are not documented.
{{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
This property is used as an alias to get password from credential providers,
or, fall back to using the value as password in clear text. There is also a
caveat that credential providers can not be a HDFS-based file system, as
mentioned in HADOOP-11934, to prevent cyclic dependency issue.
This should be documented in core-default.xml and GroupsMapping.md
{{hadoop.security.credential.clear-text-fallback}}
This property controls whether or not to fall back to storing credential
password as cleartext.
This should be documented in core-default.xml.
{{hadoop.security.credential.provider.path}}
This is mentioned in _CredentialProvider API Guide_, but not in core-default.xml
The "Supported Features" in _CredentialProvider API Guide_ should link back to
GroupsMapping.md#LDAP Groups Mapping
{{hadoop.security.credstore.java-keystore-provider.password-file}}
This is the password file to protect credential files.
was:
A few properties are not documented.
{{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
This property is used as an alias to get password from credential providers,
or, fall back to using the value as password in clear text. There is also a
caveat that credential providers can not be a HDFS-based file system, as
mentioned in HADOOP-11934, to prevent cyclic dependency issue.
This should be documented in core-default.xml and GroupsMapping.md
{{hadoop.security.credential.clear-text-fallback}}
This property controls whether or not to fall back to storing credential
password as cleartext.
This should be documented in core-default.xml.
{{hadoop.security.credential.provider.path}}
This is mentioned in _CredentialProvider API Guide_, but not in core-default.xml
The "Supported Features" in _CredentialProvider API Guide_ should link back to
GroupsMapping.md#LDAP Groups Mapping
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Priority: Minor
> Labels: documentation
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
> {{hadoop.security.credstore.java-keystore-provider.password-file}}
> This is the password file to protect credential files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13442) Optimize UGI group lookups
Daryn Sharp created HADOOP-13442:
Summary: Optimize UGI group lookups
Key: HADOOP-13442
URL: https://issues.apache.org/jira/browse/HADOOP-13442
Project: Hadoop Common
Issue Type: Improvement
Reporter: Daryn Sharp
Assignee: Daryn Sharp
{{UGI#getGroups}} and its usage is inefficient. The list is unnecessarily
converted to multiple collections.
For _every_ invocation, the {{List}} from the group provider is
converted into a {{LinkedHashSet}} (to de-dup), back to a {{String[]}}.
Then callers testing for group membership convert back to a {{List}}.
This should be done once to reduce allocations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13434) Add quoting to Shell class
[ https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HADOOP-13434: --- Status: Patch Available (was: Open) > Add quoting to Shell class > -- > > Key: HADOOP-13434 > URL: https://issues.apache.org/jira/browse/HADOOP-13434 > Project: Hadoop Common > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HADOOP-13434.patch, HADOOP-13434.patch > > > The Shell class makes assumptions that the parameters won't have spaces or > other special characters, even when it invokes bash. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13441:
-
Component/s: security
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Priority: Minor
> Labels: documentation
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13441:
-
Affects Version/s: 2.6.0
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Priority: Minor
> Labels: documentation
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13441:
-
Labels: documentation (was: )
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Priority: Minor
> Labels: documentation
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements
[
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399707#comment-15399707
]
Chris Nauroth commented on HADOOP-13403:
[~pattipaka], in addition to the earlier code review feedback, I have a
higher-level question about the implementation of the {{executeParallel}}
method.
Typical usage of a {{ThreadPoolExecutor}} is to configure it with a certain
thread count and then submit multiple tasks, with each task performing a single
isolated action. When tasks are submitted, they get added to an internal queue
(the {{LinkedBlockingQueue}} in your code). Then, the number of threads all
independently pull and execute tasks from the queue.
The {{executeParallel}} method in your current patch does not follow this
typical usage of {{ThreadPoolExecutor}}. Instead, it submits a number of tasks
identical to the thread count, and the code in each task iterates through an
array of work items ({{FileMetadata[]}}) tracked manually. The array is shared
state across all of those tasks/threads, so safe access requires additional
synchronization code.
It appears the current {{executeParallel}} reimplements functionality already
built into the {{ThreadPoolExecutor}}. Switching this code to more idiomatic
usage of {{ThreadPoolExecutor}} would make it easier for maintainers to
understand, and it would remove the need for extra cross-thread
synchronization, which is always tricky to get right. For example, assuming 10
threads configured for {{fs.azure.delete.threads}} and a deletion of 100 blobs,
I recommend using a {{ThreadPoolExecutor}} configured with 10 threads and
submitting 100 task instances to it, each task deleting a single blob, with no
shared state across the tasks/threads.
I also see the current logic carefully tracks the last exception that happened
on a thread. Maybe that's why you implemented {{executeParallel}} this way?
However, there is also a more idiomatic way to look for exceptions: call
{{ThreadPoolExecutor#submit}} for all tasks, track every returned {{Future}} in
a list, and then iterate through that list calling {{Future#get}} on each one.
If any task encountered an exception, it would propagate as an
{{ExecutionException}}, and then you can unwrap it to get the underlying
exception that was thrown on the thread.
I also see logic for handling {{RejectedExecutionException}}. In practice, I
expect this won't happen, because the {{LinkedBlockingQueue}} will cause the
caller submitting the task to block instead of failing if it runs out of
capacity. If you're concerned about this, then another way to approach this is
to look into
[{{ThreadPoolExecutor.CallerRunsPolicy}}|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.CallerRunsPolicy.html],
which would fall back to running the task in the main submitting thread.
If there is some other reason I'm missing that requires the current unusual
implementation of {{executeParallel}}, please let me know. Thank you.
> AzureNativeFileSystem rename/delete performance improvements
>
>
> Key: HADOOP-13403
> URL: https://issues.apache.org/jira/browse/HADOOP-13403
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure
>Affects Versions: 2.7.2
>Reporter: Subramanyam Pattipaka
>Assignee: Subramanyam Pattipaka
> Fix For: 2.9.0
>
> Attachments: HADOOP-13403-001.patch, HADOOP-13403-002.patch
>
>
> WASB Performance Improvements
> Problem
> ---
> Azure Native File system operations like rename/delete which has large number
> of directories and/or files in the source directory are experiencing
> performance issues. Here are possible reasons
> a)We first list all files under source directory hierarchically. This is
> a serial operation.
> b)After collecting the entire list of files under a folder, we delete or
> rename files one by one serially.
> c)There is no logging information available for these costly operations
> even in DEBUG mode leading to difficulty in understanding wasb performance
> issues.
> Proposal
> -
> Step 1: Rename and delete operations will generate a list all files under the
> source folder. We need to use azure flat listing option to get list with
> single request to azure store. We have introduced config
> fs.azure.flatlist.enable to enable this option. The default value is 'false'
> which means flat listing is disabled.
> Step 2: Create thread pool and threads dynamically based on user
> configuration. These thread pools will be deleted after operation is over.
> We are introducing introducing two new configs
> a) fs.azure.rename.threads : Config to set number of rename
> threads. Default value is 0 which means no threading.
[jira] [Created] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
Wei-Chiu Chuang created HADOOP-13441:
Summary: Document LdapGroupsMapping keystore password properties
Key: HADOOP-13441
URL: https://issues.apache.org/jira/browse/HADOOP-13441
Project: Hadoop Common
Issue Type: Bug
Reporter: Wei-Chiu Chuang
Priority: Minor
A few properties are not documented.
{{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
This property is used as an alias to get password from credential providers,
or, fall back to using the value as password in clear text. There is also a
caveat that credential providers can not be a HDFS-based file system, as
mentioned in HADOOP-11934, to prevent cyclic dependency issue.
This should be documented in core-default.xml and GroupsMapping.md
{{hadoop.security.credential.clear-text-fallback}}
This property controls whether or not to fall back to storing credential
password as cleartext.
This should be documented in core-default.xml.
{{hadoop.security.credential.provider.path}}
This is mentioned in _CredentialProvider API Guide_, but not in core-default.xml
The "Supported Features" in _CredentialProvider API Guide_ should link back to
GroupsMapping.md#LDAP Groups Mapping
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13426) More efficiently build IPC responses
[
https://issues.apache.org/jira/browse/HADOOP-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399525#comment-15399525
]
Hadoop QA commented on HADOOP-13426:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 193 unchanged - 3 fixed = 194 total (was 196) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820981/HADOOP-13426.1.patch |
| JIRA Issue | HADOOP-13426 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux afa313284062 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 204a205 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10119/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10119/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10119/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> More efficiently build IPC responses
>
>
> Key: HADOOP-13426
> URL: https://issues.apache.org/jira/browse/HADOOP-13426
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments:
[jira] [Commented] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[
https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399461#comment-15399461
]
Wei-Chiu Chuang commented on HADOOP-13353:
--
I see that when the IOException is thrown, the log message does not preserve
its stacktrace. Could you also update the fix to include the stacktrace?
Basically, change it to:
{code}
catch (IOException ioe) {
LOG.warn("Exception while trying to password for alias " + alias + ": ",
ioe);
}
{code}
> LdapGroupsMapping getPassward shouldn't return null when IOException throws
> ---
>
> Key: HADOOP-13353
> URL: https://issues.apache.org/jira/browse/HADOOP-13353
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Zhaohao Liang
> Attachments: HADOOP-13353.patch
>
>
> When IOException throws in getPassword(), getPassword() return null String,
> this will cause setConf() throws java.lang.NullPointerException when check
> isEmpty() on null string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[ https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13353: - Labels: supportability (was: ) > LdapGroupsMapping getPassward shouldn't return null when IOException throws > --- > > Key: HADOOP-13353 > URL: https://issues.apache.org/jira/browse/HADOOP-13353 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Zhaohao Liang > Labels: supportability > Attachments: HADOOP-13353.patch > > > When IOException throws in getPassword(), getPassword() return null String, > this will cause setConf() throws java.lang.NullPointerException when check > isEmpty() on null string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[ https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399455#comment-15399455 ] Wei-Chiu Chuang commented on HADOOP-13353: -- The fix looks good to me. Thanks [~liang...@gmail.com] for the contribution. Would you also like to add a regression test? > LdapGroupsMapping getPassward shouldn't return null when IOException throws > --- > > Key: HADOOP-13353 > URL: https://issues.apache.org/jira/browse/HADOOP-13353 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Zhaohao Liang > Attachments: HADOOP-13353.patch > > > When IOException throws in getPassword(), getPassword() return null String, > this will cause setConf() throws java.lang.NullPointerException when check > isEmpty() on null string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13426) More efficiently build IPC responses
[ https://issues.apache.org/jira/browse/HADOOP-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-13426: - Attachment: HADOOP-13426.1.patch Make checkstyle happy. > More efficiently build IPC responses > > > Key: HADOOP-13426 > URL: https://issues.apache.org/jira/browse/HADOOP-13426 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-13426.1.patch, HADOOP-13426.patch > > > The call response buffer is allowed to dynamically grow until a max size is > reached. Often times the full size of the response can be known in advance > which avoids copies. This is very advantageous for large responses. > Automatic framing of the response buffer will also prevent unnecessary > allocations and copies when the size is/isn't known. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399420#comment-15399420
]
Hadoop QA commented on HADOOP-9565:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-9565 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820968/HADOOP-9565-007.patch
|
| JIRA Issue | HADOOP-9565 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10118/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy
[
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399409#comment-15399409
]
Daryn Sharp commented on HADOOP-13436:
--
bq. Does it mean it leads to performance issues?
Technically yes, but if the total code path was profiled it's probably not even
a blip. I took a look at {{MultipleLinearRandomRetry}} and actually caches the
{{toString()}} value so it's not an issue with that policy.
Now the problem is making sure all current and future policies properly adhere
to a contract required to ensure connection lookups work correctly. I'd
suggest that {{RetryPolicy}} should be an abstract class rather than an
interface. This will ensure the contract is enforced.
{{RetryPolicy}} would implement {{hashCode}}, {{equals}}, and {{toString}} as
final methods using the impl of {{MultipleLinearRandomRetry}}. Except
{{toString}} would call a new abstract {{getName}} as the value it will cache.
If compatibility is an overriding concern then I guess a new abstract class,
ex. {{AbstractRetryPolicy}}, that implements {{RetryPolicy}} could be created.
Existing polices would then extend that class. This has the downside of future
policies re-introducing this jira's buggy behavior.
> RPC connections are leaking due to missing equals override in
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat}
> enabled. Internally, Client#getConnection is doing lookup relying on
> ConnectionId#equals, which composes checking Subclass-of-RetryPolicy#equals.
> If subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every
> instances of RetryPolicy with equivalent fields' values (e.g.
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy is called:
> {noformat}
> at
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
>
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pieter Reuse updated HADOOP-9565:
-
Attachment: HADOOP-9565-007.patch
Uploaded patch version 7:
* Renamed UnsatisfiedSemanticsException to UnsatisfiedFeatureException
* Renamed ObjectStoreFeatures to ObjectStoreFeature and changed to an
enum-based approach
* added style improvements to FileOutputCommitter, since we're changing a lot
of lines throughout the class. Makes this patch prone to become non-applicable
when other patches change this class.
Chris, thanks for pointing out Steve will be enjoying a well earned break the
coming weeks. We will have patience w.r.t. his response.
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13222) s3a.mkdirs() to delete empty fake parent directories
[
https://issues.apache.org/jira/browse/HADOOP-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398947#comment-15398947
]
Shen Yinjie edited comment on HADOOP-13222 at 7/29/16 8:36 AM:
---
I firstly create object {{aa/bb}} with aws-java-sdk, and it is not empty, it is
ok to create object {{aa/bb/cc/dd}} (with sdk or hadoop shell). but when excute
hadoop fs -ls and -cat with {{aa/bb}}, it will only work on the file
{{aa/bb}},though it is reasonable for s3a .
was (Author: shenyinjie):
I firstly create object {{aa/bb}} with aws-java-sdk, and it is not empty, it is
ok to create object {{aa/bb/cc/dd}} (with sdk or hadoop shell). but when excute
hadoop fs -ls and -cat with {{aa/bb}}, something wrong will appear,though it
is reasonable for s3a .
> s3a.mkdirs() to delete empty fake parent directories
>
>
> Key: HADOOP-13222
> URL: https://issues.apache.org/jira/browse/HADOOP-13222
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Priority: Minor
>
> {{S3AFileSystem.mkdirs()}} has a TODO comment: what do do about fake parent
> directories.
> The answer is: as with files, they should be deleted. This can be done
> asynchronously
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13222) s3a.mkdirs() to delete empty fake parent directories
[
https://issues.apache.org/jira/browse/HADOOP-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398947#comment-15398947
]
Shen Yinjie edited comment on HADOOP-13222 at 7/29/16 8:25 AM:
---
I firstly create object {{aa/bb}} with aws-java-sdk, and it is not empty, it is
ok to create object {{aa/bb/cc/dd}} (with sdk or hadoop shell). but when excute
hadoop fs -ls and -cat with {{aa/bb}}, something wrong will appear,though it
is reasonable for s3a .
was (Author: shenyinjie):
I firstly create object {{aa/bb}} with aws-java-sdk, and it is not empty, it is
ok to create object {{aa/bb/cc/dd}} (with sdk or hadoop shell). but when excute
hadoop fs -ls and -cat with {{aa/bb}}, something wrong will appear,though it
is reasonable in s3a itself.
> s3a.mkdirs() to delete empty fake parent directories
>
>
> Key: HADOOP-13222
> URL: https://issues.apache.org/jira/browse/HADOOP-13222
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Priority: Minor
>
> {{S3AFileSystem.mkdirs()}} has a TODO comment: what do do about fake parent
> directories.
> The answer is: as with files, they should be deleted. This can be done
> asynchronously
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13222) s3a.mkdirs() to delete empty fake parent directories
[
https://issues.apache.org/jira/browse/HADOOP-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398947#comment-15398947
]
Shen Yinjie commented on HADOOP-13222:
--
I firstly create object {{aa/bb}} with aws-java-sdk, and it is not empty, it is
ok to create object {{aa/bb/cc/dd}} (with sdk or hadoop shell). but when excute
hadoop fs -ls and -cat with {{aa/bb}}, something wrong will appear,though it
is reasonable in s3a itself.
> s3a.mkdirs() to delete empty fake parent directories
>
>
> Key: HADOOP-13222
> URL: https://issues.apache.org/jira/browse/HADOOP-13222
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Priority: Minor
>
> {{S3AFileSystem.mkdirs()}} has a TODO comment: what do do about fake parent
> directories.
> The answer is: as with files, they should be deleted. This can be done
> asynchronously
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading
[
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398851#comment-15398851
]
Hadoop QA commented on HADOOP-13437:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch
generated 2 new + 8 unchanged - 8 fixed = 10 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820891/HADOOP-13437.03.patch
|
| JIRA Issue | HADOOP-13437 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 88f6c78ebe27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8ebf2e9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10117/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10117/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10117/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.6.0
>
[jira] [Comment Edited] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading
[
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398801#comment-15398801
]
Xiao Chen edited comment on HADOOP-13437 at 7/29/16 6:29 AM:
-
Thanks [~andrew.wang] for the great comments! I confess I was too obsessed with
the fact that reloading doesn't load the properties, and overlooked the 'like a
big behavior change'.
However, with more digging, I still think this is right.
- In {{Configuration}} parsing, later entries with the same name win. So in a
{{Configuration}} object, a key will only have 1 value - the latest specified
value.
- Good call on swapping. Updated the code to do it this way - previously, the
old entries for default/whitelist are preserved after reloading. I think this
is purely a bug, because 1. looking at the latest config file you'll never
figure out the actual acls in memory. 2. no way to update/remove a previously
configured entry without restart.
Added a test case for this.
- Above said, I really don't see why the {{containsKey}} check should be kept.
I looked at HADOOP-10433, HADOOP-10758 and HADOOP-11341, but don't see it
mentioned. That seems to be a redundant check for me.
bq. Is the new "continue" case important? Not sure if we should ride over a
possibly malformed ACL file.
Not at all. It's ignoring the exception, and then depending on the fact
{{aclType != null}} to continue - functionally the same. I can leave that
untouched to reduce our change scope.
Patch 3 is attached, with more thorough testing. I will do another round of
test-case thinking tomorrow, but feel free to comment on what's here.
was (Author: xiaochen):
Thanks [~andrew.wang] for the great comments! I confess I was too obsessed with
the fact that reloading doesn't load the properties, and overlooked the 'like a
big behavior change'.
However, with more digging, I still think this is right.
- In {{Configuration}} parsing, later entries with the same name win. So in a
{{Configuration}} object, a key will only have 1 value - the latest specified
value.
- Good call on swapping. Updated the code to do it this way - previously, the
old entries for default/whitelist are preserved after reloading. I think this
is purely a bug, because 1. looking at the latest config file you'll never
figure out the actual acls in memory. 2. no way to update/remove a previously
configured entry without restart.
Added a test case for this.
- Above said, I really don't see why the {{containsKey}} check should be kept.
I looked at HADOOP-10433, HADOOP-10758 and HADOOP-11341, but don't see it
mentioned. That seems to be a redundant check for me.
bq. Is the new "continue" case important? Not sure if we should ride over a
possibly malformed ACL file.
Not at all. It's ignoring the exception, and then depending on the fact
{{aclType != null}} to continue - functionally the same. I can leave that
untouched to reduce our change scope.
Patch 4 is attached, with more thorough testing. I will do another round of
test-case thinking tomorrow, but feel free to comment on what's here.
> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch, HADOOP-13437.02.patch,
> HADOOP-13437.03.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any
> difference in behavior.
> Credit to [~dilaver] for finding this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading
[
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13437:
---
Attachment: HADOOP-13437.03.patch
Thanks [~andrew.wang] for the great comments! I confess I was too obsessed with
the fact that reloading doesn't load the properties, and overlooked the 'like a
big behavior change'.
However, with more digging, I still think this is right.
- In {{Configuration}} parsing, later entries with the same name win. So in a
{{Configuration}} object, a key will only have 1 value - the latest specified
value.
- Good call on swapping. Updated the code to do it this way - previously, the
old entries for default/whitelist are preserved after reloading. I think this
is purely a bug, because 1. looking at the latest config file you'll never
figure out the actual acls in memory. 2. no way to update/remove a previously
configured entry without restart.
Added a test case for this.
- Above said, I really don't see why the {{containsKey}} check should be kept.
I looked at HADOOP-10433, HADOOP-10758 and HADOOP-11341, but don't see it
mentioned. That seems to be a redundant check for me.
bq. Is the new "continue" case important? Not sure if we should ride over a
possibly malformed ACL file.
Not at all. It's ignoring the exception, and then depending on the fact
{{aclType != null}} to continue - functionally the same. I can leave that
untouched to reduce our change scope.
Patch 4 is attached, with more thorough testing. I will do another round of
test-case thinking tomorrow, but feel free to comment on what's here.
> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch, HADOOP-13437.02.patch,
> HADOOP-13437.03.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any
> difference in behavior.
> Credit to [~dilaver] for finding this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13440) FileContext does not react on changing umask via configuration
[
https://issues.apache.org/jira/browse/HADOOP-13440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398790#comment-15398790
]
Hadoop QA commented on HADOOP-13440:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820693/YARN-5425.00.patch |
| JIRA Issue | HADOOP-13440 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0dd28a8b30d0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8ebf2e9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10116/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10116/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FileContext does not react on changing umask via configuration
> --
>
> Key: HADOOP-13440
> URL: https://issues.apache.org/jira/browse/HADOOP-13440
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Akira Ajisaka
> Attachments:
