[jira] [Commented] (HADOOP-13168) Support Future.get with timeout in ipc async calls
[ https://issues.apache.org/jira/browse/HADOOP-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290455#comment-15290455 ] Xiaobing Zhou commented on HADOOP-13168: Thank you working on this. Some comments: 1. AsyncGet For doc of AsyncGet#get, would 'When timeout == 0' better be 'When timeout = 0'? @return doc should be added for Util#asyncGetTimeout2WaitTimeout In Util#asyncGetTimeout2WaitTimeout, it returns 1 ms, is it possible to lead to overhead given huge number of calls? Can we use a smaller time unit? Object#wait(long timeout, int nanos) could be used in that case. 2. ipc.Client In ipc.Client#call, make #get override has the same signature as AsyncGet#get, e.g. AsyncGet#get(long timeout, TimeUnit unit) throws IOException, TimeoutException, InterruptedException If getRpcResponse throws IOException, there's no chance to change done back 3. ProtobufRpcEngine Similarly, make #get override has the same signature as AsyncGet#get > Support Future.get with timeout in ipc async calls > -- > > Key: HADOOP-13168 > URL: https://issues.apache.org/jira/browse/HADOOP-13168 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: c13168_20160517.patch, c13168_20160518.patch > > > Currently, the Future returned by ipc async call only support Future.get() > but not Future.get(timeout, unit). We should support the latter as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290451#comment-15290451 ] Andrew Wang commented on HADOOP-12893: -- I'm working through the above, thanks for the review Billie. I've been looking at [HBase's NOTICE|https://github.com/apache/hbase/blob/master/NOTICE.txt] as an example. [~busbey], hope you don't mind if I ask some questions? * The Hadoop NOTICE entry. [The guidelines|http://www.apache.org/dev/licensing-howto.html#bundle-asf-product] say you don't need to duplicate this normally? * The Bootstrap and Guava NOTICE entries. They are both ASLv2 and don't have their own NOTICE files. Are these notices required? I ask these because [the guidelines|http://www.apache.org/dev/licensing-howto.html#mod-notice] say: "Do not add anything to NOTICE which is not legally required." So far I've seen a huge variety of NOTICE styles looking through Hadoop's deps, and I'm still looking for a canonical example of what this is supposed to look like. > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-12893.01.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13132) LoadBalancingKMSClientProvider ClassCastException on AuthenticationException
[
https://issues.apache.org/jira/browse/HADOOP-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13132:
-
Attachment: HADOOP-13132.003.patch
v03: I looked at a similar jira (HADOOP-13180) and finally understood why it is
possible to throw {{AuthenticationException}} in
{{KMSClientProvider#decryptEncryptedKey}}, which can only throw IOException and
GeneralSecurityException:
there is a dirty trick in the code that cheats Java compiler and which throws
AuthenticationException as an unchecked RuntimeException! Then, in
{{LoadBalancingKMSClientProvider#doOp}}, because {{AuthenticationException}} is
not a {{RuntimeException}} subclass, it is rethrown in a {{WrapperException}},
which causes the type cast in
{{LoadBalancingKMSClientProvider#decryptEncryptedKey}} to fail. The v3 patch
reproduces how it is done. (More specifically, the exception is thrown in
{{HttpExceptionUtils#validateResponse}}).
Please review again, thanks!
> LoadBalancingKMSClientProvider ClassCastException on AuthenticationException
>
>
> Key: HADOOP-13132
> URL: https://issues.apache.org/jira/browse/HADOOP-13132
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Miklos Szurap
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-13132.001.patch, HADOOP-13132.002.patch,
> HADOOP-13132.003.patch
>
>
> An Oozie job with a single shell action fails (may not be important, but if
> you needs the exact details I can provide them) with an error message coming
> from NodeManager:
> {code}
> 2016-05-10 11:10:14,290 ERROR
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[LogAggregationService #652,5,main] threw an Exception.
> java.lang.ClassCastException:
> org.apache.hadoop.security.authentication.client.AuthenticationException
> cannot be cast to java.security.GeneralSecurityException
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:189)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1521)
> at org.apache.hadoop.fs.Hdfs.createInternal(Hdfs.java:108)
> at org.apache.hadoop.fs.Hdfs.createInternal(Hdfs.java:59)
> at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:683)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:679)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:679)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogWriter$1.run(AggregatedLogFormat.java:382)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogWriter$1.run(AggregatedLogFormat.java:377)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogWriter.(AggregatedLogFormat.java:376)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.uploadLogsForContainers(AppLogAggregatorImpl.java:246)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.doAppLogAggregation(AppLogAggregatorImpl.java:456)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.run(AppLogAggregatorImpl.java:421)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$2.run(LogAggregationService.java:384)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The unsafe cast is here:
> https://github.com/apache/hadoop/blob/2e1d0ff4e901b8313c8d71869735b94ed8bc40a0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/LoadBalancingKMSClientProvider.java#L189
> Because of this ClassCastException:
> - an uncaught exception is raised
> - we do not see the exact "caused by" exception/message
> - the oozie job fails
> - YARN logs are not reported/saved
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail:
[jira] [Created] (HADOOP-13180) Encryption Zone data Run mr with execption:AuthenticationException can't be found in cache
lushuai created HADOOP-13180: Summary: Encryption Zone data Run mr with execption:AuthenticationException can't be found in cache Key: HADOOP-13180 URL: https://issues.apache.org/jira/browse/HADOOP-13180 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.1 Reporter: lushuai org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (owner=hive, renewer=yarn, realUser=, issueDate=1463627282514, maxDate=1464232082514, sequenceNumber=217, masterKeyId=2) can't be found in cache at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:249) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:622) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:566) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:675) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (owner=hive, renewer=yarn, realUser=, issueDate=1463627282514, maxDate=1464232082514, sequenceNumber=217, masterKeyId=2) can't be found in cache at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:487) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:445) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:719) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1347) at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1446) at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1431) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:801) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:261) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:246) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-12723) S3A: Add ability to plug in any AWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290358#comment-15290358
]
Steven Wong commented on HADOOP-12723:
--
It's the "Missing package-info.java file" again.
> S3A: Add ability to plug in any AWSCredentialsProvider
> --
>
> Key: HADOOP-12723
> URL: https://issues.apache.org/jira/browse/HADOOP-12723
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Steven Wong
>Assignee: Steven Wong
> Attachments: HADOOP-12723.0.patch, HADOOP-12723.1.patch,
> HADOOP-12723.2.patch, HADOOP-12723.3.patch, HADOOP-12723.4.patch,
> HADOOP-12723.5.patch
>
>
> Although S3A currently has built-in support for
> {{org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider}},
> {{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, and
> {{org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider}}, it does not
> support any other credentials provider that implements the
> {{com.amazonaws.auth.AWSCredentialsProvider}} interface. Supporting the
> ability to plug in any {{com.amazonaws.auth.AWSCredentialsProvider}} instance
> will expand the options for S3 credentials, such as:
> * temporary credentials from STS, e.g. via
> {{com.amazonaws.auth.STSSessionCredentialsProvider}}
> * IAM role-based credentials, e.g. via
> {{com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider}}
> * a custom credentials provider that satisfies one's own needs, e.g.
> bucket-specific credentials, user-specific credentials, etc.
> To support this, we can add a configuration for the fully qualified class
> name of a credentials provider, to be loaded by
> {{S3AFileSystem.initialize(URI, Configuration)}}.
> The configured credentials provider should implement
> {{com.amazonaws.auth.AWSCredentialsProvider}} and have a constructor that
> accepts {{(URI uri, Configuration conf)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[
https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290308#comment-15290308
]
Hadoop QA commented on HADOOP-13174:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-common-project: The patch generated 1 new + 141
unchanged - 0 fixed = 142 total (was 141) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 40s {color}
| {color:red} hadoop-auth in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 3s {color} |
{color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.security.authentication.server.TestAuthenticationFilter |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804842/HADOOP-13174.03.patch
|
| JIRA Issue | HADOOP-13174 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6ffe4fad24b9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9499/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
|
| unit |
[jira] [Updated] (HADOOP-13179) TooRunner is not thread-safe because commons-cli OptionBuilder is not thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongbin ma updated HADOOP-13179:
Attachment: HADOOP-13179-master.patch
> TooRunner is not thread-safe because commons-cli OptionBuilder is not
> thread-safe
> -
>
> Key: HADOOP-13179
> URL: https://issues.apache.org/jira/browse/HADOOP-13179
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: hongbin ma
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13179-master.patch
>
>
> I'm running into similar issues like
> http://stackoverflow.com/questions/22462665/is-hadoops-toorunner-thread-safe,
> the author's observation seem to make sense to me. However when I checked the
> hadoop github trunk I found the issue still not fixed.
> Chris Nauroth further investigated this issue, here's his quote:
> {quote}
> The root cause is that commons-cli OptionBuilder is not thread-safe.
> https://commons.apache.org/proper/commons-cli/apidocs/org/apache/commons/cl
> i/OptionBuilder.html
> According to this issue, commons-cli doesn't plan to change that and
> instead chose to document the lack of thread-safety.
> https://issues.apache.org/jira/browse/CLI-209
> I think we can solve this in Hadoop, probably with a one-line change to
> make GenericOptionsParser#buildGeneralOptions a synchronized method.
> {quote}
> I'll soon upload a patch for this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13179) TooRunner is not thread-safe because commons-cli OptionBuilder is not thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongbin ma updated HADOOP-13179:
Fix Version/s: 2.8.0
Status: Patch Available (was: Open)
> TooRunner is not thread-safe because commons-cli OptionBuilder is not
> thread-safe
> -
>
> Key: HADOOP-13179
> URL: https://issues.apache.org/jira/browse/HADOOP-13179
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: hongbin ma
>Priority: Minor
> Fix For: 2.8.0
>
>
> I'm running into similar issues like
> http://stackoverflow.com/questions/22462665/is-hadoops-toorunner-thread-safe,
> the author's observation seem to make sense to me. However when I checked the
> hadoop github trunk I found the issue still not fixed.
> Chris Nauroth further investigated this issue, here's his quote:
> {quote}
> The root cause is that commons-cli OptionBuilder is not thread-safe.
> https://commons.apache.org/proper/commons-cli/apidocs/org/apache/commons/cl
> i/OptionBuilder.html
> According to this issue, commons-cli doesn't plan to change that and
> instead chose to document the lack of thread-safety.
> https://issues.apache.org/jira/browse/CLI-209
> I think we can solve this in Hadoop, probably with a one-line change to
> make GenericOptionsParser#buildGeneralOptions a synchronized method.
> {quote}
> I'll soon upload a patch for this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[
https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290283#comment-15290283
]
Hadoop QA commented on HADOOP-13130:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 54s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s {color} | {color:green} root: The patch generated 0 new + 36 unchanged - 7
fixed = 36 total (was 43) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 46s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s {color} |
{color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color}
[jira] [Commented] (HADOOP-13145) In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
[
https://issues.apache.org/jira/browse/HADOOP-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290280#comment-15290280
]
Hadoop QA commented on HADOOP-13145:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
57s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s
{color} | {color:red} root: The patch generated 1 new + 31 unchanged - 0 fixed
= 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 9s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 11s
{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s
{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch
|
| JIRA Issue | HADOOP-13145 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 971586d256d2 3.13.0-36-lowlatency #63-Ubuntu
[jira] [Created] (HADOOP-13179) TooRunner is not thread-safe because commons-cli OptionBuilder is not thread-safe
hongbin ma created HADOOP-13179:
---
Summary: TooRunner is not thread-safe because commons-cli
OptionBuilder is not thread-safe
Key: HADOOP-13179
URL: https://issues.apache.org/jira/browse/HADOOP-13179
Project: Hadoop Common
Issue Type: Bug
Reporter: hongbin ma
Priority: Minor
I'm running into similar issues like
http://stackoverflow.com/questions/22462665/is-hadoops-toorunner-thread-safe,
the author's observation seem to make sense to me. However when I checked the
hadoop github trunk I found the issue still not fixed.
Chris Nauroth further investigated this issue, here's his quote:
{quote}
The root cause is that commons-cli OptionBuilder is not thread-safe.
https://commons.apache.org/proper/commons-cli/apidocs/org/apache/commons/cl
i/OptionBuilder.html
According to this issue, commons-cli doesn't plan to change that and
instead chose to document the lack of thread-safety.
https://issues.apache.org/jira/browse/CLI-209
I think we can solve this in Hadoop, probably with a one-line change to
make GenericOptionsParser#buildGeneralOptions a synchronized method.
{quote}
I'll soon upload a patch for this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12723) S3A: Add ability to plug in any AWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290258#comment-15290258
]
Hadoop QA commented on HADOOP-12723:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 56s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 23s
{color} | {color:red} root: The patch generated 1 new + 18 unchanged - 0 fixed
= 19 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 52s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 16s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804818/HADOOP-12723.5.patch |
| JIRA Issue | HADOOP-12723 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 9f5e00809753 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
|
| Console output |
[jira] [Commented] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
[
https://issues.apache.org/jira/browse/HADOOP-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290235#comment-15290235
]
Hadoop QA commented on HADOOP-13177:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 10s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804782/HADOOP-13177.001.patch
|
| JIRA Issue | HADOOP-13177 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux d0bca2d9897f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9497/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9497/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to
> include libhadoop.dylib.
> ---
>
> Key: HADOOP-13177
> URL: https://issues.apache.org/jira/browse/HADOOP-13177
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13177.001.patch
>
>
> OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
> dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test
> configuration, then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[jira] [Commented] (HADOOP-13044) Amazon S3 library 10.10.60+ (JDK8u60+) depends on http components 4.3
[
https://issues.apache.org/jira/browse/HADOOP-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290230#comment-15290230
]
Kai Sasaki commented on HADOOP-13044:
-
[~ste...@apache.org] Thanks for taking care!
> Amazon S3 library 10.10.60+ (JDK8u60+) depends on http components 4.3
> -
>
> Key: HADOOP-13044
> URL: https://issues.apache.org/jira/browse/HADOOP-13044
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
>Affects Versions: 2.8.0
> Environment: JDK 8u60
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Attachments: HADOOP-13044.01.patch
>
>
> In case of using AWS SDK in the classpath of hadoop, we faced an issue caused
> by incompatiblity of AWS SDK and httpcomponents.
> {code}
> java.lang.NoSuchFieldError: INSTANCE
> at
> com.amazonaws.http.conn.SdkConnectionKeepAliveStrategy.getKeepAliveDuration(SdkConnectionKeepAliveStrategy.java:48)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:535)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> {code}
> The latest AWS SDK depends on 4.3.x which has
> [DefaultConnectionKeepAliveStrategy.INSTANCE|http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/org/apache/http/impl/client/DefaultConnectionKeepAliveStrategy.html#INSTANCE].
> This field is introduced from 4.3.
> This will allow us to avoid {{CLASSPATH}} confliction around httpclient
> versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[
https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13174:
---
Attachment: HADOOP-13174.03.patch
Thank you [~yzhangal] for the review!
Patch 3 addresses most of your comments, except:
1. My intention is to only log when the token is updated by {{token.set}}.
3. Added request and token, but response doesn't have a {{toString}}, so not
logged here. I think we can have enough information since caller will have
response code logged.
> Add more debug logs for delegation tokens and authentication
>
>
> Key: HADOOP-13174
> URL: https://issues.apache.org/jira/browse/HADOOP-13174
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HADOOP-13174.01.patch, HADOOP-13174.02.patch,
> HADOOP-13174.03.patch
>
>
> Recently I debugged several authentication related problems, and found that
> the debug logs are not enough to identify a problem.
> This jira improves it by adding more debug/trace logs along the line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[
https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290122#comment-15290122
]
Yongjun Zhang commented on HADOOP-13174:
Hi [~xiaochen],
Thanks for reporting the issue and working on the patch. I have a few comments
here:
1. AuthenticatedURL:
{{LOG.trace}} is added for {{if}} condition {{if (value.length() > 0) {}}, do
we need to trace the {{else}} condition?
2. AuthenticatedURL. For all LOG.trace added to this class, suggest to add
{{respCode}}'s value to the trace log message, so to distinguish the cases.
3. AuthenticationFilter:
{code}
LOG.debug("managementOperation returned false.");
{code}
suggest to include the parameter value passed to managementOperation {{token,
httpRequest, httpResponse}}
4. KMCClientProvider
{code}
LOG.trace("Creating connection from url: {}", url);
{code}
suggest to add {{method}} parameter to the log
5. KMCClientProvider
Suggest to add {{conn}} to the first trace log of {{call}}, plus
{{expectedResponse}}, {{authRetryCount}}
Thanks.
> Add more debug logs for delegation tokens and authentication
>
>
> Key: HADOOP-13174
> URL: https://issues.apache.org/jira/browse/HADOOP-13174
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HADOOP-13174.01.patch, HADOOP-13174.02.patch
>
>
> Recently I debugged several authentication related problems, and found that
> the debug logs are not enough to identify a problem.
> This jira improves it by adding more debug/trace logs along the line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13145) In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
[
https://issues.apache.org/jira/browse/HADOOP-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290079#comment-15290079
]
Chris Nauroth commented on HADOOP-13145:
Interestingly, you're getting a much slower run than me for S3A and a much
faster run than me for WASB. I'm in the US Pacific Northwest. My S3 bucket is
in US-west-2. My Azure Storage account is in West US.
{code}
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.389 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 143.99 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
{code}
bq. Could it be made one of the scaleable tests where it takes a config of
option on scale so can be made configurable?
We definitely could do that, but in my test runs, the large file tests don't
show a significantly longer execution time. (See below for my timings.) Are
the large file tests a long haul in your environment?
Maybe a more effective change would be to cut down the number of test cases. I
could keep just {{deepDirectoryStructureToRemote}}, {{largeFilesToRemote}},
{{deepDirectoryStructureFromRemote}} and {{largeFilesFromRemote}}. If I do
that, then my S3A execution time comes down to 90 seconds. I don't think it
sacrifices much in terms of coverage.
Let me know your thoughts, and then I'll update the patch.
{code}
{code}
> In DistCp, prevent unnecessary getFileStatus call when not preserving
> metadata.
> ---
>
> Key: HADOOP-13145
> URL: https://issues.apache.org/jira/browse/HADOOP-13145
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13145.001.patch, HADOOP-13145.003.patch
>
>
> After DistCp copies a file, it calls {{getFileStatus}} to get the
> {{FileStatus}} from the destination so that it can compare to the source and
> update metadata if necessary. If the DistCp command was run without the
> option to preserve metadata attributes, then this additional
> {{getFileStatus}} call is wasteful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12723) S3A: Add ability to plug in any AWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290072#comment-15290072
]
Steven Wong commented on HADOOP-12723:
--
HADOOP-12723.5.patch is tested against S3 Standard from us-east-1, including
running all TestS3A* unit tests.
> S3A: Add ability to plug in any AWSCredentialsProvider
> --
>
> Key: HADOOP-12723
> URL: https://issues.apache.org/jira/browse/HADOOP-12723
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Steven Wong
>Assignee: Steven Wong
> Attachments: HADOOP-12723.0.patch, HADOOP-12723.1.patch,
> HADOOP-12723.2.patch, HADOOP-12723.3.patch, HADOOP-12723.4.patch,
> HADOOP-12723.5.patch
>
>
> Although S3A currently has built-in support for
> {{org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider}},
> {{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, and
> {{org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider}}, it does not
> support any other credentials provider that implements the
> {{com.amazonaws.auth.AWSCredentialsProvider}} interface. Supporting the
> ability to plug in any {{com.amazonaws.auth.AWSCredentialsProvider}} instance
> will expand the options for S3 credentials, such as:
> * temporary credentials from STS, e.g. via
> {{com.amazonaws.auth.STSSessionCredentialsProvider}}
> * IAM role-based credentials, e.g. via
> {{com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider}}
> * a custom credentials provider that satisfies one's own needs, e.g.
> bucket-specific credentials, user-specific credentials, etc.
> To support this, we can add a configuration for the fully qualified class
> name of a credentials provider, to be loaded by
> {{S3AFileSystem.initialize(URI, Configuration)}}.
> The configured credentials provider should implement
> {{com.amazonaws.auth.AWSCredentialsProvider}} and have a constructor that
> accepts {{(URI uri, Configuration conf)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12723) S3A: Add ability to plug in any AWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Wong updated HADOOP-12723:
-
Attachment: HADOOP-12723.5.patch
[~steve_l], thanks for taking time to review the patch.
Attaching HADOOP-12723.5.patch that accepts all your suggestions, except for #2
which I'm unclear about what you meant.
Regarding HADOOP-12537, an earlier comment of mine (above) compares it with an
alternative with different functionality (short-lived vs. long-lived
credentials). But, yes, one can implement HADOOP-12537 using HADOOP-12723:
Implement and plug in a new provider that, just like HADOOP-12537 currently
does, looks up the session token from configuration and returns a
com.amazonaws.auth.BasicSessionCredentials.
Regarding what other providers I have in mind, I have seen a use case for a
provider that doles out credentials using
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider with a role ARN
(specified in a config property), and a use case for doling out different
role-based credentials based on the S3 bucket being accessed (using config
properties that map bucket names to role ARNs).
> S3A: Add ability to plug in any AWSCredentialsProvider
> --
>
> Key: HADOOP-12723
> URL: https://issues.apache.org/jira/browse/HADOOP-12723
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Steven Wong
>Assignee: Steven Wong
> Attachments: HADOOP-12723.0.patch, HADOOP-12723.1.patch,
> HADOOP-12723.2.patch, HADOOP-12723.3.patch, HADOOP-12723.4.patch,
> HADOOP-12723.5.patch
>
>
> Although S3A currently has built-in support for
> {{org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider}},
> {{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, and
> {{org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider}}, it does not
> support any other credentials provider that implements the
> {{com.amazonaws.auth.AWSCredentialsProvider}} interface. Supporting the
> ability to plug in any {{com.amazonaws.auth.AWSCredentialsProvider}} instance
> will expand the options for S3 credentials, such as:
> * temporary credentials from STS, e.g. via
> {{com.amazonaws.auth.STSSessionCredentialsProvider}}
> * IAM role-based credentials, e.g. via
> {{com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider}}
> * a custom credentials provider that satisfies one's own needs, e.g.
> bucket-specific credentials, user-specific credentials, etc.
> To support this, we can add a configuration for the fully qualified class
> name of a credentials provider, to be loaded by
> {{S3AFileSystem.initialize(URI, Configuration)}}.
> The configured credentials provider should implement
> {{com.amazonaws.auth.AWSCredentialsProvider}} and have a constructor that
> accepts {{(URI uri, Configuration conf)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13174: --- Attachment: HADOOP-13174.02.patch Patch 2 to fix the checkstyle about imports. IMHO the line that's over 80 is more readable in one line than split into 2, so I left that as-is. > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Attachments: HADOOP-13174.01.patch, HADOOP-13174.02.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9613) [JDK8] Update jersey version to latest 1.x release
[
https://issues.apache.org/jira/browse/HADOOP-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289998#comment-15289998
]
Wangda Tan commented on HADOOP-9613:
Thanks [~ozawa] for this great effort!
I just tried YARN-3368 with trunk applied this patch. I didn't see any critical
issues, the only issue I can see is, when no application submitted/running on
the cluster, applications page cannot be loaded. Probably caused by the reason
mentioned above:
bq. I found one incompatible change about Jersey - after upgrading from 1.12 to
1.13, the root element whose content is empty collection is changed from null
to empty object({}). Related to following change:
https://java.net/jira/browse/JERSEY-1168
Other pages work fine. So from YARN-3368's perspective, this patch is safe to
be merged. [~sunilg], could you share your thoughts as well?
> [JDK8] Update jersey version to latest 1.x release
> --
>
> Key: HADOOP-9613
> URL: https://issues.apache.org/jira/browse/HADOOP-9613
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 2.4.0, 3.0.0-alpha1
>Reporter: Timothy St. Clair
>Assignee: Tsuyoshi Ozawa
> Labels: UpgradeKeyLibrary, maven
> Attachments: HADOOP-2.2.0-9613.patch,
> HADOOP-9613.004.incompatible.patch, HADOOP-9613.005.incompatible.patch,
> HADOOP-9613.006.incompatible.patch, HADOOP-9613.007.incompatible.patch,
> HADOOP-9613.008.incompatible.patch, HADOOP-9613.009.incompatible.patch,
> HADOOP-9613.010.incompatible.patch, HADOOP-9613.011.incompatible.patch,
> HADOOP-9613.012.incompatible.patch, HADOOP-9613.013.incompatible.patch,
> HADOOP-9613.014.incompatible.patch, HADOOP-9613.014.incompatible.patch,
> HADOOP-9613.015.incompatible.patch, HADOOP-9613.016.incompatible.patch,
> HADOOP-9613.017.incompatible.patch, HADOOP-9613.1.patch, HADOOP-9613.2.patch,
> HADOOP-9613.3.patch, HADOOP-9613.patch
>
>
> Update pom.xml dependencies exposed when running a mvn-rpmbuild against
> system dependencies on Fedora 18.
> The existing version is 1.8 which is quite old.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13140) FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes
[
https://issues.apache.org/jira/browse/HADOOP-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289987#comment-15289987
]
Mingliang Liu commented on HADOOP-13140:
Thanks [~cmccabe] for your review and commit.
> FileSystem#initialize must not attempt to create StorageStatistics objects
> with null or empty schemes
> -
>
> Key: HADOOP-13140
> URL: https://issues.apache.org/jira/browse/HADOOP-13140
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13140.000.patch, HADOOP-13140.001.patch,
> HADOOP-13140.002.patch
>
>
> {{org.apache.hadoop.fs.GlobalStorageStatistics#put}} is not checking the null
> scheme, and the internal map will complain NPE. This was reported by a flaky
> test {{TestFileSystemApplicationHistoryStore}}. Thanks [~brahmareddy] for
> reporting.
> To address this,
> # Fix the test by providing a valid URI, e.g. {{file:///}}
> # Guard the null scheme in {{GlobalStorageStatistics#put}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13160) Suppress checkstyle JavadocPackage check for test source
[ https://issues.apache.org/jira/browse/HADOOP-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289956#comment-15289956 ] John Zhuge commented on HADOOP-13160: - Thanks [~boky01] for the suggestion and testing on Windows. > Suppress checkstyle JavadocPackage check for test source > > > Key: HADOOP-13160 > URL: https://issues.apache.org/jira/browse/HADOOP-13160 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HADOOP-13160.001.patch, HADOOP-13160.002.patch, > HADOOP-13160.003.patch > > > Suppress "Missing package-info.java" checkstyle error for test source files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13157) Follow-on improvements to hadoop credential commands
[
https://issues.apache.org/jira/browse/HADOOP-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13157:
-
Resolution: Fixed
Fix Version/s: (was: 2.9.0)
2.8.0
Status: Resolved (was: Patch Available)
Thanks Mike for the branch-2* patches, and to Akira for catching my mistake
with the branch-2 commit. Committed respective patches to branch-2 and 2.8.
> Follow-on improvements to hadoop credential commands
>
>
> Key: HADOOP-13157
> URL: https://issues.apache.org/jira/browse/HADOOP-13157
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0
>Reporter: Mike Yoder
>Assignee: Mike Yoder
> Fix For: 2.8.0
>
> Attachments: HADOOP-13157.001.patch, HADOOP-13157.002.patch,
> HADOOP-13157.003.branch-2.8.patch, HADOOP-13157.004.branch-2.patch
>
>
> [~andrew.wang] had some follow-up code review comments from HADOOP-12942.
> Hence this issue.
> Ping [~lmccay] as well.
> The comments:
> {quote}
> Overall this looks okay, the only correctness question I have is about the
> difference in behavior when the pwfile doesn't exist.
> The rest are all nits, would be nice to do these cleanups though.
> File
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java:
> Line 147:
> Could this be a static helper?
> Line 161: new
> The javadoc says it returns null in this situation. This is also a difference
> from the implementation in the AbstractJKSP. Intentional?
> Line 175: private void locateKeystore() throws IOException {
> static helper? for the construct*Path methods too?
> File
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyShell.java:
> Line 50: @VisibleForTesting public static final String NO_VALID_PROVIDERS =
> FYI for the future, our coding style is to put annotations on their own
> separate line.
> File
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java:
> Line 326: private char[] locatePassword() throws IOException {
> this method looks very similar to the one in JavaKeyStoreProvider, except the
> env var it looks for is different, is there potential for code reuse?
> Line 394: "o In the environment variable " +
> Using a "*" is the usual way of doing a bullet point, e.g. markdown and wiki
> syntax.
> Line 399:
> "http://hadoop.apache.org/docs/current/hadoop-project-dist/; +
> This link is not tied to a version, so could be inaccurate.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9613) [JDK8] Update jersey version to latest 1.x release
[ https://issues.apache.org/jira/browse/HADOOP-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289909#comment-15289909 ] Sangjin Lee commented on HADOOP-9613: - Thanks [~gtCarrera9]! > [JDK8] Update jersey version to latest 1.x release > -- > > Key: HADOOP-9613 > URL: https://issues.apache.org/jira/browse/HADOOP-9613 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.4.0, 3.0.0-alpha1 >Reporter: Timothy St. Clair >Assignee: Tsuyoshi Ozawa > Labels: UpgradeKeyLibrary, maven > Attachments: HADOOP-2.2.0-9613.patch, > HADOOP-9613.004.incompatible.patch, HADOOP-9613.005.incompatible.patch, > HADOOP-9613.006.incompatible.patch, HADOOP-9613.007.incompatible.patch, > HADOOP-9613.008.incompatible.patch, HADOOP-9613.009.incompatible.patch, > HADOOP-9613.010.incompatible.patch, HADOOP-9613.011.incompatible.patch, > HADOOP-9613.012.incompatible.patch, HADOOP-9613.013.incompatible.patch, > HADOOP-9613.014.incompatible.patch, HADOOP-9613.014.incompatible.patch, > HADOOP-9613.015.incompatible.patch, HADOOP-9613.016.incompatible.patch, > HADOOP-9613.017.incompatible.patch, HADOOP-9613.1.patch, HADOOP-9613.2.patch, > HADOOP-9613.3.patch, HADOOP-9613.patch > > > Update pom.xml dependencies exposed when running a mvn-rpmbuild against > system dependencies on Fedora 18. > The existing version is 1.8 which is quite old. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
[
https://issues.apache.org/jira/browse/HADOOP-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289884#comment-15289884
]
Chris Nauroth commented on HADOOP-13177:
Haha, thanks. :-) I'll commit after I get a pre-commit response.
> Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to
> include libhadoop.dylib.
> ---
>
> Key: HADOOP-13177
> URL: https://issues.apache.org/jira/browse/HADOOP-13177
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13177.001.patch
>
>
> OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
> dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test
> configuration, then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[
https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289883#comment-15289883
]
Chris Nauroth commented on HADOOP-13130:
Thank you for patch 008. This is looking good. I completed a clean parallel
test run in ~26 minutes against S3 buckets in US-west-2.
Thank you for the clarification on {{eventually}} and {{Callable}}. That
makes sense.
There is one thing left unaddressed from my last comment. There was an EOF
assertion that used a descriptive message, and I was wondering if you wanted to
do the same for other EOF assertions. I'm not sure if this slipped through the
cracks in 008, or if it was a conscious choice not to change it.
To make sure it's clear, here is the descriptive assertion:
{code}
assertEquals("Expected EOF got char " + (char) c, -1, c);
{code}
...and here are the other ones that might benefit from the same message...
{code}
assertEquals(-1, instream.read(buf));
{code}
{code}
assertEquals(-1, instream.read(instream.getPos(), buf, 0, buf.length));
{code}
{code}
assertEquals(-1, instream.read(shortLen + 510, buf, 0, buf.length));
{code}
> s3a failures can surface as RTEs, not IOEs
> --
>
> Key: HADOOP-13130
> URL: https://issues.apache.org/jira/browse/HADOOP-13130
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch,
> HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch,
> HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch,
> HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch
>
>
> S3A failures happening in the AWS library surface as
> {{AmazonClientException}} derivatives, rather than IOEs. As the amazon
> exceptions are runtime exceptions, any code which catches IOEs for error
> handling breaks.
> The fix will be to catch and wrap. The hard thing will be to wrap it with
> meaningful exceptions rather than a generic IOE. Furthermore, if anyone has
> been catching AWS exceptions, they are going to be disappointed. That means
> that fixing this situation could be considered "incompatible" —but only for
> code which contains assumptions about the underlying FS and the exceptions
> they raise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9613) [JDK8] Update jersey version to latest 1.x release
[ https://issues.apache.org/jira/browse/HADOOP-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289833#comment-15289833 ] Li Lu commented on HADOOP-9613: --- BTW has anyone taken a look at the -1s sent by Jenkins? Any quick notes w.r.t the checkstyle warnings? > [JDK8] Update jersey version to latest 1.x release > -- > > Key: HADOOP-9613 > URL: https://issues.apache.org/jira/browse/HADOOP-9613 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.4.0, 3.0.0-alpha1 >Reporter: Timothy St. Clair >Assignee: Tsuyoshi Ozawa > Labels: UpgradeKeyLibrary, maven > Attachments: HADOOP-2.2.0-9613.patch, > HADOOP-9613.004.incompatible.patch, HADOOP-9613.005.incompatible.patch, > HADOOP-9613.006.incompatible.patch, HADOOP-9613.007.incompatible.patch, > HADOOP-9613.008.incompatible.patch, HADOOP-9613.009.incompatible.patch, > HADOOP-9613.010.incompatible.patch, HADOOP-9613.011.incompatible.patch, > HADOOP-9613.012.incompatible.patch, HADOOP-9613.013.incompatible.patch, > HADOOP-9613.014.incompatible.patch, HADOOP-9613.014.incompatible.patch, > HADOOP-9613.015.incompatible.patch, HADOOP-9613.016.incompatible.patch, > HADOOP-9613.017.incompatible.patch, HADOOP-9613.1.patch, HADOOP-9613.2.patch, > HADOOP-9613.3.patch, HADOOP-9613.patch > > > Update pom.xml dependencies exposed when running a mvn-rpmbuild against > system dependencies on Fedora 18. > The existing version is 1.8 which is quite old. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9613) [JDK8] Update jersey version to latest 1.x release
[ https://issues.apache.org/jira/browse/HADOOP-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289825#comment-15289825 ] Li Lu commented on HADOOP-9613: --- Tried the latest incompatible patch with the latest YARN-2928 branch. There are a few trivial patch apply failures but easily fixable (since the YARN-2928 branch is not up-to-date to the latest trunk). Tried all UTs introduced in the branch and with my local deployment. All things seems to be fine. cc/[~sjlee0] > [JDK8] Update jersey version to latest 1.x release > -- > > Key: HADOOP-9613 > URL: https://issues.apache.org/jira/browse/HADOOP-9613 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.4.0, 3.0.0-alpha1 >Reporter: Timothy St. Clair >Assignee: Tsuyoshi Ozawa > Labels: UpgradeKeyLibrary, maven > Attachments: HADOOP-2.2.0-9613.patch, > HADOOP-9613.004.incompatible.patch, HADOOP-9613.005.incompatible.patch, > HADOOP-9613.006.incompatible.patch, HADOOP-9613.007.incompatible.patch, > HADOOP-9613.008.incompatible.patch, HADOOP-9613.009.incompatible.patch, > HADOOP-9613.010.incompatible.patch, HADOOP-9613.011.incompatible.patch, > HADOOP-9613.012.incompatible.patch, HADOOP-9613.013.incompatible.patch, > HADOOP-9613.014.incompatible.patch, HADOOP-9613.014.incompatible.patch, > HADOOP-9613.015.incompatible.patch, HADOOP-9613.016.incompatible.patch, > HADOOP-9613.017.incompatible.patch, HADOOP-9613.1.patch, HADOOP-9613.2.patch, > HADOOP-9613.3.patch, HADOOP-9613.patch > > > Update pom.xml dependencies exposed when running a mvn-rpmbuild against > system dependencies on Fedora 18. > The existing version is 1.8 which is quite old. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13178) TestShellBasedIdMapping.testStaticMapUpdate doesn't work on OS X
[
https://issues.apache.org/jira/browse/HADOOP-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289827#comment-15289827
]
Allen Wittenauer commented on HADOOP-13178:
---
{code}
testStaticMapUpdate(org.apache.hadoop.security.TestShellBasedIdMapping) Time
elapsed: 2.12 sec <<< FAILURE!
java.lang.AssertionError: expected:<> but was:<-1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.security.TestShellBasedIdMapping.testStaticMapUpdate(TestShellBasedIdMapping.java:198)
{code}
> TestShellBasedIdMapping.testStaticMapUpdate doesn't work on OS X
>
>
> Key: HADOOP-13178
> URL: https://issues.apache.org/jira/browse/HADOOP-13178
> Project: Hadoop Common
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>
> TestShellBasedIdMapping.testStaticMapUpdate throws an error on OS X.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13178) TestShellBasedIdMapping.testStaticMapUpdate doesn't work on OS X
Allen Wittenauer created HADOOP-13178: - Summary: TestShellBasedIdMapping.testStaticMapUpdate doesn't work on OS X Key: HADOOP-13178 URL: https://issues.apache.org/jira/browse/HADOOP-13178 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer TestShellBasedIdMapping.testStaticMapUpdate throws an error on OS X. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13160) Suppress checkstyle JavadocPackage check for test source
[
https://issues.apache.org/jira/browse/HADOOP-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289812#comment-15289812
]
Andras Bokor commented on HADOOP-13160:
---
+1
[^HADOOP-13160.003.patch] works on Windows. {code}"src\\test.*missing
package-"{code} pattern was found 394 times before the patch and 0 matches
after the patch.
> Suppress checkstyle JavadocPackage check for test source
>
>
> Key: HADOOP-13160
> URL: https://issues.apache.org/jira/browse/HADOOP-13160
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-13160.001.patch, HADOOP-13160.002.patch,
> HADOOP-13160.003.patch
>
>
> Suppress "Missing package-info.java" checkstyle error for test source files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289809#comment-15289809
]
Jason Lowe commented on HADOOP-12990:
-
I agree with [~qwertymaniac] that the right way to fix this is to add a
separate codec intended to be compatible with the LZ4 CLI tools. I don't think
it would be too hard to accomplish. We'd need to avoid using
BlockCompressorStream/BlockDecompressorStream since those are doing the
big-endian lengths that will totally confuse the CLI tools. Instead we need to
do the same little-endian block-length logic done by the CLI tool along with
the 4-byte of zeros for the end marker. Seems to simply be a 4-byte
little-endian length per block, but sometimes the high bit can be set? Haven't
looked at the lz4 CLI code to see what the exact logic is.
Bonus points for including the xxhash code so Hadoop can generate and validate
checksums.
> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
> Issue Type: Bug
> Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13160) Suppress checkstyle JavadocPackage check for test source
[
https://issues.apache.org/jira/browse/HADOOP-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289807#comment-15289807
]
Andras Bokor commented on HADOOP-13160:
---
+1
[^HADOOP-13160.003.patch] works on Windows. {{src\\test.*missing package-}}
pattern was found 394 times before the patch and 0 matches after the patch.
> Suppress checkstyle JavadocPackage check for test source
>
>
> Key: HADOOP-13160
> URL: https://issues.apache.org/jira/browse/HADOOP-13160
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-13160.001.patch, HADOOP-13160.002.patch,
> HADOOP-13160.003.patch
>
>
> Suppress "Missing package-info.java" checkstyle error for test source files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
[
https://issues.apache.org/jira/browse/HADOOP-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289803#comment-15289803
]
Allen Wittenauer commented on HADOOP-13177:
---
+1, given I'm already running this patch. lol
> Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to
> include libhadoop.dylib.
> ---
>
> Key: HADOOP-13177
> URL: https://issues.apache.org/jira/browse/HADOOP-13177
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13177.001.patch
>
>
> OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
> dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test
> configuration, then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
[
https://issues.apache.org/jira/browse/HADOOP-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13177:
---
Priority: Minor (was: Major)
> Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to
> include libhadoop.dylib.
> ---
>
> Key: HADOOP-13177
> URL: https://issues.apache.org/jira/browse/HADOOP-13177
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13177.001.patch
>
>
> OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
> dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test
> configuration, then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
[
https://issues.apache.org/jira/browse/HADOOP-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13177:
---
Attachment: HADOOP-13177.001.patch
I'm attaching a patch with just the Surefire configuration changes spun out of
HDFS-3296.
> Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to
> include libhadoop.dylib.
> ---
>
> Key: HADOOP-13177
> URL: https://issues.apache.org/jira/browse/HADOOP-13177
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13177.001.patch
>
>
> OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
> dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test
> configuration, then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
[
https://issues.apache.org/jira/browse/HADOOP-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13177:
---
Status: Patch Available (was: Open)
> Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to
> include libhadoop.dylib.
> ---
>
> Key: HADOOP-13177
> URL: https://issues.apache.org/jira/browse/HADOOP-13177
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13177.001.patch
>
>
> OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
> dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test
> configuration, then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13177) Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib.
Chris Nauroth created HADOOP-13177:
--
Summary: Native tests fail on OS X, because DYLD_LIBRARY_PATH is
not defined to include libhadoop.dylib.
Key: HADOOP-13177
URL: https://issues.apache.org/jira/browse/HADOOP-13177
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Chris Nauroth
Assignee: Chris Nauroth
OS X uses {{DYLD_LIBRARY_PATH}} instead of {{LD_LIBRARY_PATH}} to control the
dynamic link path. If we set {{DYLD_LIBRARY_PATH}} in the test configuration,
then we can run native tests on Mac.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13156) create-release.sh doesn't work for branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan resolved HADOOP-13156.
-
Resolution: Duplicate
This is duplicated to HADOOP-12892
> create-release.sh doesn't work for branch-2.8
> -
>
> Key: HADOOP-13156
> URL: https://issues.apache.org/jira/browse/HADOOP-13156
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Wangda Tan
>Priority: Blocker
>
> A couple of issues found while trying to run dev-support/create-release.sh:
> 1) Missing files like release-notes.html and CHANGE.txt
> 2) After remove lines to copy release-notes.html/CHANGE.txt, still saw some
> issues, for example:
> {code}
> usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
>cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ...
> target_directory
> Failed! running cp -r ../../target/r2.8.0-SNAPSHOT/api
> ../../target/r2.8.0-SNAPSHOT/css
> ../../target/r2.8.0-SNAPSHOT/dependency-analysis.html
> ../../target/r2.8.0-SNAPSHOT/hadoop-annotations
> ../../target/r2.8.0-SNAPSHOT/hadoop-ant
> ../../target/r2.8.0-SNAPSHOT/hadoop-archive-logs
> ../../target/r2.8.0-SNAPSHOT/hadoop-archives
> ../../target/r2.8.0-SNAPSHOT/hadoop-assemblies
> ../../target/r2.8.0-SNAPSHOT/hadoop-auth
> ../../target/r2.8.0-SNAPSHOT/hadoop-auth-examples
> ../../target/r2.8.0-SNAPSHOT/hadoop-aws
> ../../target/r2.8.0-SNAPSHOT/hadoop-azure
> ../../target/r2.8.0-SNAPSHOT/hadoop-common-project
> ../../target/r2.8.0-SNAPSHOT/hadoop-datajoin
> ../../target/r2.8.0-SNAPSHOT/hadoop-dist
> ../../target/r2.8.0-SNAPSHOT/hadoop-distcp
> ../../target/r2.8.0-SNAPSHOT/hadoop-extras
> ../../target/r2.8.0-SNAPSHOT/hadoop-gridmix
> ../../target/r2.8.0-SNAPSHOT/hadoop-hdfs-bkjournal
> ../../target/r2.8.0-SNAPSHOT/hadoop-hdfs-httpfs
> ../../target/r2.8.0-SNAPSHOT/hadoop-hdfs-nfs
> ../../target/r2.8.0-SNAPSHOT/hadoop-hdfs-project
> ../../target/r2.8.0-SNAPSHOT/hadoop-kms
> ../../target/r2.8.0-SNAPSHOT/hadoop-mapreduce
> ../../target/r2.8.0-SNAPSHOT/hadoop-mapreduce-client
> ../../target/r2.8.0-SNAPSHOT/hadoop-mapreduce-examples
> ../../target/r2.8.0-SNAPSHOT/hadoop-maven-plugins
> ../../target/r2.8.0-SNAPSHOT/hadoop-minicluster
> ../../target/r2.8.0-SNAPSHOT/hadoop-minikdc
> ../../target/r2.8.0-SNAPSHOT/hadoop-nfs
> ../../target/r2.8.0-SNAPSHOT/hadoop-openstack
> ../../target/r2.8.0-SNAPSHOT/hadoop-pipes
> ../../target/r2.8.0-SNAPSHOT/hadoop-project-dist
> ../../target/r2.8.0-SNAPSHOT/hadoop-rumen
> ../../target/r2.8.0-SNAPSHOT/hadoop-sls
> ../../target/r2.8.0-SNAPSHOT/hadoop-streaming
> ../../target/r2.8.0-SNAPSHOT/hadoop-tools
> ../../target/r2.8.0-SNAPSHOT/hadoop-yarn
> ../../target/r2.8.0-SNAPSHOT/hadoop-yarn-project
> ../../target/r2.8.0-SNAPSHOT/images ../../target/r2.8.0-SNAPSHOT/index.html
> ../../target/r2.8.0-SNAPSHOT/project-reports.html
> hadoop-2.8.0-SNAPSHOT/share/doc/hadoop/ in
> /Users/wtan/sandbox/hadoop-erie-copy/hadoop-dist/target
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13168) Support Future.get with timeout in ipc async calls
[ https://issues.apache.org/jira/browse/HADOOP-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HADOOP-13168: - Attachment: c13168_20160518.patch c13168_20160518.patch: revises javadoc and some other minor changes. > Support Future.get with timeout in ipc async calls > -- > > Key: HADOOP-13168 > URL: https://issues.apache.org/jira/browse/HADOOP-13168 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: c13168_20160517.patch, c13168_20160518.patch > > > Currently, the Future returned by ipc async call only support Future.get() > but not Future.get(timeout, unit). We should support the latter as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13145) In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
[
https://issues.apache.org/jira/browse/HADOOP-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289674#comment-15289674
]
Steve Loughran commented on HADOOP-13145:
-
tested -003 against s3 ireland and azure.
{code}
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 223.843 sec -
in org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
...
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.354 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
{code}
Interesting how much faster azure is.
The patch, is, as it stands, it's going to add 4 min to a TestS3A* test
pattern. Could it be made one of the scaleable tests where it takes a config of
option on scale so can be made configurable? There are already some tests which
use {{scale.test.operation.count}} to control scale; we could have one on
distcp file size, with the large file size being driven by it. Make it
something in KB and it could easily be tuned for those of us in a different
country from an S3 endpoint.
> In DistCp, prevent unnecessary getFileStatus call when not preserving
> metadata.
> ---
>
> Key: HADOOP-13145
> URL: https://issues.apache.org/jira/browse/HADOOP-13145
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13145.001.patch, HADOOP-13145.003.patch
>
>
> After DistCp copies a file, it calls {{getFileStatus}} to get the
> {{FileStatus}} from the destination so that it can compare to the source and
> update metadata if necessary. If the DistCp command was run without the
> option to preserve metadata attributes, then this additional
> {{getFileStatus}} call is wasteful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13065) Add a new interface for retrieving FS and FC Statistics
[ https://issues.apache.org/jira/browse/HADOOP-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289689#comment-15289689 ] Ming Ma commented on HADOOP-13065: -- [~liuml07] [~cmccabe], yeah the perf could be an issue. Regarding moving it to ReadStatistics, the problem is there is no easy way for MR framework to get hold of the HDFS streams used by the application. > Add a new interface for retrieving FS and FC Statistics > --- > > Key: HADOOP-13065 > URL: https://issues.apache.org/jira/browse/HADOOP-13065 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Ram Venkatesh >Assignee: Mingliang Liu > Fix For: 2.8.0 > > Attachments: HADOOP-13065-007.patch, HADOOP-13065.008.patch, > HADOOP-13065.009.patch, HADOOP-13065.010.patch, HADOOP-13065.011.patch, > HADOOP-13065.012.patch, HADOOP-13065.013.patch, HDFS-10175.000.patch, > HDFS-10175.001.patch, HDFS-10175.002.patch, HDFS-10175.003.patch, > HDFS-10175.004.patch, HDFS-10175.005.patch, HDFS-10175.006.patch, > TestStatisticsOverhead.java > > > Currently FileSystem.Statistics exposes the following statistics: > BytesRead > BytesWritten > ReadOps > LargeReadOps > WriteOps > These are in-turn exposed as job counters by MapReduce and other frameworks. > There is logic within DfsClient to map operations to these counters that can > be confusing, for instance, mkdirs counts as a writeOp. > Proposed enhancement: > Add a statistic for each DfsClient operation including create, append, > createSymlink, delete, exists, mkdirs, rename and expose them as new > properties on the Statistics object. The operation-specific counters can be > used for analyzing the load imposed by a particular job on HDFS. > For example, we can use them to identify jobs that end up creating a large > number of files. > Once this information is available in the Statistics object, the app > frameworks like MapReduce can expose them as additional counters to be > aggregated and recorded as part of job summary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13145) In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
[
https://issues.apache.org/jira/browse/HADOOP-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13145:
Status: Patch Available (was: Open)
> In DistCp, prevent unnecessary getFileStatus call when not preserving
> metadata.
> ---
>
> Key: HADOOP-13145
> URL: https://issues.apache.org/jira/browse/HADOOP-13145
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13145.001.patch, HADOOP-13145.003.patch
>
>
> After DistCp copies a file, it calls {{getFileStatus}} to get the
> {{FileStatus}} from the destination so that it can compare to the source and
> update metadata if necessary. If the DistCp command was run without the
> option to preserve metadata attributes, then this additional
> {{getFileStatus}} call is wasteful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13137) TraceAdmin should support Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13137:
-
Summary: TraceAdmin should support Kerberized cluster (was: TraceAdmin
should support Kerberized NameNode)
> TraceAdmin should support Kerberized cluster
>
>
> Key: HADOOP-13137
> URL: https://issues.apache.org/jira/browse/HADOOP-13137
> Project: Hadoop Common
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.6.0, 3.0.0-alpha1
> Environment: CDH5.5.1 cluster with Kerberos
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: Kerberos
> Attachments: HADOOP-13137.001.patch, HADOOP-13137.002.patch
>
>
> When I run {{hadoop trace}} command for a Kerberized NameNode, it failed with
> the following error:
> [hdfs@weichiu-encryption-1 root]$ hadoop trace -list -host
> weichiu-encryption-1.vpc.cloudera.com:802216/05/12 00:02:13 WARN ipc.Client:
> Exception encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 16/05/12 00:02:13 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@vpc.cloudera.com (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: java.lang.IllegalArgumentException: Failed to specify
> server's Kerberos principal name; Host Details : local host is:
> "weichiu-encryption-1.vpc.cloudera.com/172.26.8.185"; destination host is:
> "weichiu-encryption-1.vpc.cloudera.com":8022;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1470)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy11.listSpanReceivers(Unknown Source)
> at
> org.apache.hadoop.tracing.TraceAdminProtocolTranslatorPB.listSpanReceivers(TraceAdminProtocolTranslatorPB.java:58)
> at
> org.apache.hadoop.tracing.TraceAdmin.listSpanReceivers(TraceAdmin.java:68)
> at org.apache.hadoop.tracing.TraceAdmin.run(TraceAdmin.java:177)
> at org.apache.hadoop.tracing.TraceAdmin.main(TraceAdmin.java:195)
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1519)
> at org.apache.hadoop.ipc.Client.call(Client.java:1442)
> ... 7 more
> Caused by: java.lang.IllegalArgumentException: Failed to specify server's
> Kerberos principal name
> at
> org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:322)
> at
> org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:231)
> at
> org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
> ... 10 more
> It is failing because {{TraceAdmin}} does not set up the property
> {{CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_USER_NAME_KEY}}
> Fixing it may require some restructuring, as the NameNode principal
> {{dfs.namenode.kerberos.principal}} is a HDFS property, but TraceAdmin is in
> hadoop-common. Or, specify it with a new command {{-principal}}. Any
> suggestions? Thanks
[jira] [Updated] (HADOOP-13137) TraceAdmin should support Kerberized NameNode
[
https://issues.apache.org/jira/browse/HADOOP-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13137:
-
Status: Patch Available (was: Open)
Submit the patch for precommit test.
[~cmccabe], [~steve_l], would you mind to review the patch? Thank you very much!
> TraceAdmin should support Kerberized NameNode
> -
>
> Key: HADOOP-13137
> URL: https://issues.apache.org/jira/browse/HADOOP-13137
> Project: Hadoop Common
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.6.0, 3.0.0-alpha1
> Environment: CDH5.5.1 cluster with Kerberos
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: Kerberos
> Attachments: HADOOP-13137.001.patch, HADOOP-13137.002.patch
>
>
> When I run {{hadoop trace}} command for a Kerberized NameNode, it failed with
> the following error:
> [hdfs@weichiu-encryption-1 root]$ hadoop trace -list -host
> weichiu-encryption-1.vpc.cloudera.com:802216/05/12 00:02:13 WARN ipc.Client:
> Exception encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 16/05/12 00:02:13 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@vpc.cloudera.com (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: java.lang.IllegalArgumentException: Failed to specify
> server's Kerberos principal name; Host Details : local host is:
> "weichiu-encryption-1.vpc.cloudera.com/172.26.8.185"; destination host is:
> "weichiu-encryption-1.vpc.cloudera.com":8022;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1470)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy11.listSpanReceivers(Unknown Source)
> at
> org.apache.hadoop.tracing.TraceAdminProtocolTranslatorPB.listSpanReceivers(TraceAdminProtocolTranslatorPB.java:58)
> at
> org.apache.hadoop.tracing.TraceAdmin.listSpanReceivers(TraceAdmin.java:68)
> at org.apache.hadoop.tracing.TraceAdmin.run(TraceAdmin.java:177)
> at org.apache.hadoop.tracing.TraceAdmin.main(TraceAdmin.java:195)
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1519)
> at org.apache.hadoop.ipc.Client.call(Client.java:1442)
> ... 7 more
> Caused by: java.lang.IllegalArgumentException: Failed to specify server's
> Kerberos principal name
> at
> org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:322)
> at
> org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:231)
> at
> org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
> ... 10 more
> It is failing because {{TraceAdmin}} does not set up the property
> {{CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_USER_NAME_KEY}}
> Fixing it may require some restructuring, as the NameNode principal
> {{dfs.namenode.kerberos.principal}} is a HDFS property, but TraceAdmin is in
> hadoop-common. Or, specify it with a new
[jira] [Updated] (HADOOP-13137) TraceAdmin should support Kerberized NameNode
[
https://issues.apache.org/jira/browse/HADOOP-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13137:
-
Attachment: HADOOP-13137.002.patch
v02: added a test case to test {{hadoop trace}} command against a Kerberized
cluster. In addition, updated _Enabling Dapper-like Tracing in Hadoop_ doc for
the usage.
> TraceAdmin should support Kerberized NameNode
> -
>
> Key: HADOOP-13137
> URL: https://issues.apache.org/jira/browse/HADOOP-13137
> Project: Hadoop Common
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.6.0, 3.0.0-alpha1
> Environment: CDH5.5.1 cluster with Kerberos
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: Kerberos
> Attachments: HADOOP-13137.001.patch, HADOOP-13137.002.patch
>
>
> When I run {{hadoop trace}} command for a Kerberized NameNode, it failed with
> the following error:
> [hdfs@weichiu-encryption-1 root]$ hadoop trace -list -host
> weichiu-encryption-1.vpc.cloudera.com:802216/05/12 00:02:13 WARN ipc.Client:
> Exception encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 16/05/12 00:02:13 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@vpc.cloudera.com (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: java.lang.IllegalArgumentException: Failed to specify
> server's Kerberos principal name; Host Details : local host is:
> "weichiu-encryption-1.vpc.cloudera.com/172.26.8.185"; destination host is:
> "weichiu-encryption-1.vpc.cloudera.com":8022;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1470)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy11.listSpanReceivers(Unknown Source)
> at
> org.apache.hadoop.tracing.TraceAdminProtocolTranslatorPB.listSpanReceivers(TraceAdminProtocolTranslatorPB.java:58)
> at
> org.apache.hadoop.tracing.TraceAdmin.listSpanReceivers(TraceAdmin.java:68)
> at org.apache.hadoop.tracing.TraceAdmin.run(TraceAdmin.java:177)
> at org.apache.hadoop.tracing.TraceAdmin.main(TraceAdmin.java:195)
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1519)
> at org.apache.hadoop.ipc.Client.call(Client.java:1442)
> ... 7 more
> Caused by: java.lang.IllegalArgumentException: Failed to specify server's
> Kerberos principal name
> at
> org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:322)
> at
> org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:231)
> at
> org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
> ... 10 more
> It is failing because {{TraceAdmin}} does not set up the property
> {{CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_USER_NAME_KEY}}
> Fixing it may require some restructuring, as the NameNode principal
> {{dfs.namenode.kerberos.principal}} is a HDFS property, but TraceAdmin is in
[jira] [Commented] (HADOOP-13160) Suppress checkstyle JavadocPackage check for test source
[
https://issues.apache.org/jira/browse/HADOOP-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289680#comment-15289680
]
Andras Bokor commented on HADOOP-13160:
---
When I suggested {code}{code} I tested it on Windows. But let me do
a try with the full [^HADOOP-13160.003.patch]. I will get back to here.
> Suppress checkstyle JavadocPackage check for test source
>
>
> Key: HADOOP-13160
> URL: https://issues.apache.org/jira/browse/HADOOP-13160
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-13160.001.patch, HADOOP-13160.002.patch,
> HADOOP-13160.003.patch
>
>
> Suppress "Missing package-info.java" checkstyle error for test source files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[
https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13130:
Status: Patch Available (was: Open)
> s3a failures can surface as RTEs, not IOEs
> --
>
> Key: HADOOP-13130
> URL: https://issues.apache.org/jira/browse/HADOOP-13130
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch,
> HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch,
> HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch,
> HADOOP-13130-branch-2-007.patch
>
>
> S3A failures happening in the AWS library surface as
> {{AmazonClientException}} derivatives, rather than IOEs. As the amazon
> exceptions are runtime exceptions, any code which catches IOEs for error
> handling breaks.
> The fix will be to catch and wrap. The hard thing will be to wrap it with
> meaningful exceptions rather than a generic IOE. Furthermore, if anyone has
> been catching AWS exceptions, they are going to be disappointed. That means
> that fixing this situation could be considered "incompatible" —but only for
> code which contains assumptions about the underlying FS and the exceptions
> they raise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[
https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13130:
Status: Open (was: Patch Available)
> s3a failures can surface as RTEs, not IOEs
> --
>
> Key: HADOOP-13130
> URL: https://issues.apache.org/jira/browse/HADOOP-13130
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch,
> HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch,
> HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch,
> HADOOP-13130-branch-2-007.patch
>
>
> S3A failures happening in the AWS library surface as
> {{AmazonClientException}} derivatives, rather than IOEs. As the amazon
> exceptions are runtime exceptions, any code which catches IOEs for error
> handling breaks.
> The fix will be to catch and wrap. The hard thing will be to wrap it with
> meaningful exceptions rather than a generic IOE. Furthermore, if anyone has
> been catching AWS exceptions, they are going to be disappointed. That means
> that fixing this situation could be considered "incompatible" —but only for
> code which contains assumptions about the underlying FS and the exceptions
> they raise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-003.patch
Patch 003; address checkstyle and javadoc.
This patch is rebased onto HADOOP-13130 patch 008; as such it doesn't really
merit review until that patch is in...consider it a WiP for now.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13172) And an UnsupportedFeatureException for Filesystems to throw on unsupported operations
[
https://issues.apache.org/jira/browse/HADOOP-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289646#comment-15289646
]
Steve Loughran commented on HADOOP-13172:
-
Right. In HADOOP-13130 I Was throwing {{UnsupportedOperationException}}; Chris
flagged up that that would have been a different error than an IOE, which
things often expect to catch, and which all the current impls throw. I'd
forgotten about HADOOP-12726, though I had picked up the same pattern
# we should have an explicit way to distinguish unsupported-ness from any other
IOE.
# In HADOOP-12726, [~templedf] has a patch to move all of the IOEs to
{{UnsupportedOperationException}}
I think it comes down to: what do callers expect the method to throw if its not
there. Most seem to be throwing generic IOEs, which is fundamentally not that
useful in distinguishing "unimplemented" to "not there".
Maybe lets return to that discussion, as it would be where the fs spec would
need clarification too.
> And an UnsupportedFeatureException for Filesystems to throw on unsupported
> operations
> -
>
> Key: HADOOP-13172
> URL: https://issues.apache.org/jira/browse/HADOOP-13172
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Filesystems which don't support things like append() tend to throw a simple
> IOE, which makes it hard to distinguish "append didn't work for some IO
> problem" and "append isn't implemented".
> If we add a new exception, {{UnsupportedFeatureException}}, make it the
> strict failure mode of such operations if not supported and patch all our
> filesystems to raise it, then at least code has a straightforward check. Same
> for any other unimplemented feature
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13160) Suppress checkstyle JavadocPackage check for test source
[ https://issues.apache.org/jira/browse/HADOOP-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289643#comment-15289643 ] Steve Loughran commented on HADOOP-13160: - LGTM. Has anyone tested this on windows? > Suppress checkstyle JavadocPackage check for test source > > > Key: HADOOP-13160 > URL: https://issues.apache.org/jira/browse/HADOOP-13160 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HADOOP-13160.001.patch, HADOOP-13160.002.patch, > HADOOP-13160.003.patch > > > Suppress "Missing package-info.java" checkstyle error for test source files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13044) Amazon S3 library 10.10.60+ (JDK8u60+) depends on http components 4.3
[
https://issues.apache.org/jira/browse/HADOOP-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289632#comment-15289632
]
Steve Loughran commented on HADOOP-13044:
-
the patch in HADOOP-12767 is a superset of this; closing as duplicate
> Amazon S3 library 10.10.60+ (JDK8u60+) depends on http components 4.3
> -
>
> Key: HADOOP-13044
> URL: https://issues.apache.org/jira/browse/HADOOP-13044
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
>Affects Versions: 2.8.0
> Environment: JDK 8u60
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Attachments: HADOOP-13044.01.patch
>
>
> In case of using AWS SDK in the classpath of hadoop, we faced an issue caused
> by incompatiblity of AWS SDK and httpcomponents.
> {code}
> java.lang.NoSuchFieldError: INSTANCE
> at
> com.amazonaws.http.conn.SdkConnectionKeepAliveStrategy.getKeepAliveDuration(SdkConnectionKeepAliveStrategy.java:48)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:535)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> {code}
> The latest AWS SDK depends on 4.3.x which has
> [DefaultConnectionKeepAliveStrategy.INSTANCE|http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/org/apache/http/impl/client/DefaultConnectionKeepAliveStrategy.html#INSTANCE].
> This field is introduced from 4.3.
> This will allow us to avoid {{CLASSPATH}} confliction around httpclient
> versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12767) update apache httpclient version to 4.5.2; httpcore to 4.4.4
[ https://issues.apache.org/jira/browse/HADOOP-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12767: Summary: update apache httpclient version to 4.5.2; httpcore to 4.4.4 (was: update apache httpclient version to the latest 4.5 for security) > update apache httpclient version to 4.5.2; httpcore to 4.4.4 > > > Key: HADOOP-12767 > URL: https://issues.apache.org/jira/browse/HADOOP-12767 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.7.2 >Reporter: Artem Aliev >Assignee: Artem Aliev > Attachments: HADOOP-12767-branch-2-005.patch, > HADOOP-12767-branch-2.004.patch, HADOOP-12767.001.patch, > HADOOP-12767.002.patch, HADOOP-12767.003.patch, HADOOP-12767.004.patch > > > Various SSL security fixes are needed. See: CVE-2012-6153, CVE-2011-4461, > CVE-2014-3577, CVE-2015-5262. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13044) Amazon S3 library 10.10.60+ (JDK8u60+) depends on http components 4.3
[
https://issues.apache.org/jira/browse/HADOOP-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13044:
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> Amazon S3 library 10.10.60+ (JDK8u60+) depends on http components 4.3
> -
>
> Key: HADOOP-13044
> URL: https://issues.apache.org/jira/browse/HADOOP-13044
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
>Affects Versions: 2.8.0
> Environment: JDK 8u60
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Attachments: HADOOP-13044.01.patch
>
>
> In case of using AWS SDK in the classpath of hadoop, we faced an issue caused
> by incompatiblity of AWS SDK and httpcomponents.
> {code}
> java.lang.NoSuchFieldError: INSTANCE
> at
> com.amazonaws.http.conn.SdkConnectionKeepAliveStrategy.getKeepAliveDuration(SdkConnectionKeepAliveStrategy.java:48)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:535)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> {code}
> The latest AWS SDK depends on 4.3.x which has
> [DefaultConnectionKeepAliveStrategy.INSTANCE|http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/org/apache/http/impl/client/DefaultConnectionKeepAliveStrategy.html#INSTANCE].
> This field is introduced from 4.3.
> This will allow us to avoid {{CLASSPATH}} confliction around httpclient
> versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13173) SecurityUtil.buildtokenService broke token authentication in multi-home network env
[ https://issues.apache.org/jira/browse/HADOOP-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289588#comment-15289588 ] Yuren Wu commented on HADOOP-13173: --- Looking through the code and i think in this particular case I think the best place to setup the value for useIp is in DFSUtil.createKeyProvider method This static method has visibility of job confiig and can easily setup the boolean value . Given KMS is a generic service to provide keys, the best place to handle this is to take the parameter into KMSClientProvider and cascading this flag until it reach the point to call SecurityUtil. However, that will affect too many APIs. At this moment the KMS is used by HDFS . therefore it can be done using this single fix. The fix code is in the attachment. > SecurityUtil.buildtokenService broke token authentication in multi-home > network env > --- > > Key: HADOOP-13173 > URL: https://issues.apache.org/jira/browse/HADOOP-13173 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 > Environment: multi-homed network cluster on SLES 11 >Reporter: Yuren Wu > > Hadoop cluster: multihomed network with KMS and Kerberos. > Sympton: mapreduce mapper will fail with kerberos error. (stack trace later) > Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET. > When mapreduce job submitted to the cluster from an outside ndoe (node only > has visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET > ip address. Such token in Credential's token list using IP will not be > correct resolved in the cluster. Therefore, we are using > hadoop.security.token.service.use_ip=false in the client side core-site.xml > file. > However, once job is accepted by yarn, the mapper container still throws > kerberos error. A detailed trace shows that when DFSClient is calling > decryptEncryptedDataencryptionKey function, it will reach to > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection > method. This method will call SecurityUtil.buildtokenService to obtain the > service name and use it to lookup the token in Credentials.getToken(service) > call. This call returned the service in the IP:PORT format. Therefore, no > token can be retrieved from credential's token hashmap. > After read SecurityUtil class, I think this class does not load configuration > from configuration file at all. It seems to me that this class create a > default configuration object and the useIpForTokenService variable is always > set default value true.. > I do not have enough experience to provide a fix. In my mind, somewhere we > need to provide the correct value of useIpForTokenService based on job setup. > Stack trace provided here. this is from wordcount job from example. > 2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775) > at >
[jira] [Commented] (HADOOP-13176) testDanglingLink fails on OS X
[
https://issues.apache.org/jira/browse/HADOOP-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289573#comment-15289573
]
Allen Wittenauer commented on HADOOP-13176:
---
Related test failures:
{code}
testDanglingLink(org.apache.hadoop.fs.TestSymlinkLocalFSFileContext) Time
elapsed: 0.226 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[admin]> but was:<[staff]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.TestSymlinkLocalFS.testDanglingLink(TestSymlinkLocalFS.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Tests run: 63, Failures: 1, Errors: 0, Skipped: 7, Time elapsed: 10.236 sec <<<
FAILURE! - in org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
testDanglingLink(org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem) Time
elapsed: 0.097 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[admin]> but was:<[staff]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.TestSymlinkLocalFS.testDanglingLink(TestSymlinkLocalFS.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> testDanglingLink fails on OS X
> --
>
> Key: HADOOP-13176
> URL: https://issues.apache.org/jira/browse/HADOOP-13176
> Project: Hadoop Common
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>
> This test assumes that the list of groups is given in a certain order. This
> is not the case on at least OS X 10.9.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[
https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13130:
Status: Patch Available (was: Open)
> s3a failures can surface as RTEs, not IOEs
> --
>
> Key: HADOOP-13130
> URL: https://issues.apache.org/jira/browse/HADOOP-13130
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch,
> HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch,
> HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch,
> HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch
>
>
> S3A failures happening in the AWS library surface as
> {{AmazonClientException}} derivatives, rather than IOEs. As the amazon
> exceptions are runtime exceptions, any code which catches IOEs for error
> handling breaks.
> The fix will be to catch and wrap. The hard thing will be to wrap it with
> meaningful exceptions rather than a generic IOE. Furthermore, if anyone has
> been catching AWS exceptions, they are going to be disappointed. That means
> that fixing this situation could be considered "incompatible" —but only for
> code which contains assumptions about the underlying FS and the exceptions
> they raise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13176) testDanglingLink fails on OS X
[ https://issues.apache.org/jira/browse/HADOOP-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13176: -- Description: This test assumes that the list of groups is given in a certain order. This is not the case on at least OS X 10.9, at least when the given user is an admin user. (was: This test assumes that the list of groups is given in a certain order. This is not the case on at least OS X 10.9.) > testDanglingLink fails on OS X > -- > > Key: HADOOP-13176 > URL: https://issues.apache.org/jira/browse/HADOOP-13176 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer > > This test assumes that the list of groups is given in a certain order. This > is not the case on at least OS X 10.9, at least when the given user is an > admin user. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13176) testDanglingLink fails on OS X
Allen Wittenauer created HADOOP-13176: - Summary: testDanglingLink fails on OS X Key: HADOOP-13176 URL: https://issues.apache.org/jira/browse/HADOOP-13176 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer This test assumes that the list of groups is given in a certain order. This is not the case on at least OS X 10.9. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[
https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13130:
Attachment: HADOOP-13130-branch-2-008.patch
patch 008. Address some checkstyle issues
> s3a failures can surface as RTEs, not IOEs
> --
>
> Key: HADOOP-13130
> URL: https://issues.apache.org/jira/browse/HADOOP-13130
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch,
> HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch,
> HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch,
> HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch
>
>
> S3A failures happening in the AWS library surface as
> {{AmazonClientException}} derivatives, rather than IOEs. As the amazon
> exceptions are runtime exceptions, any code which catches IOEs for error
> handling breaks.
> The fix will be to catch and wrap. The hard thing will be to wrap it with
> meaningful exceptions rather than a generic IOE. Furthermore, if anyone has
> been catching AWS exceptions, they are going to be disappointed. That means
> that fixing this situation could be considered "incompatible" —but only for
> code which contains assumptions about the underlying FS and the exceptions
> they raise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289565#comment-15289565 ] Chris Nauroth commented on HADOOP-12635: [~bograd], I'm not aware of any work in progress on append support for page blobs. If you're interested in that, then I'd recommend filing a new JIRA to track it. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: New Feature > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: Append API.docx, HADOOP-12635-004.patch, > HADOOP-12635.001.patch, HADOOP-12635.002.patch, HADOOP-12635.003.patch, > HADOOP-12635.005.patch, HADOOP-12635.006.patch, HADOOP-12635.007.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[
https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289564#comment-15289564
]
Hadoop QA commented on HADOOP-13175:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s {color} | {color:green} root: The patch generated 0 new + 0 unchanged - 44
fixed = 0 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hadoop-tools-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 32m 29s
{color} | {color:green} hadoop-tools in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s
{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804714/HADOOP-13175.001.patch
|
| JIRA Issue | HADOOP-13175 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs
[jira] [Commented] (HADOOP-13162) Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs
[
https://issues.apache.org/jira/browse/HADOOP-13162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289553#comment-15289553
]
Steve Loughran commented on HADOOP-13162:
-
you should think about including rename here, as on s3 rename is about mkdirs,
copies and deletes.
consider a sequence like
{code}
mkdir d1/d2/d3/
touch file d1/d2/f.txt
assert is dir d1/d2
assert is dir d1/d2/d3
touch d1/d2/d3/f2.txt
assert is dir d1/d2
assert is dir d1/d2/d3
rename d1/d2/d3 d1/d4
assert is dir d1/d4
assert exists d1/d4/f2.txt
rename d1/d2 s1/d4
assert is dir d1/d4
assert is dir d1/d4/d2
assert exists d1/d4/d2/f2.txt
{code}
> Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs
> ---
>
> Key: HADOOP-13162
> URL: https://issues.apache.org/jira/browse/HADOOP-13162
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HADOOP-13162-branch-2-002.patch, HADOOP-13162.001.patch
>
>
> getFileStatus is relatively expensive call and mkdirs invokes it multiple
> times depending on how deep the directory structure is. It would be good to
> reduce the number of getFileStatus calls in such cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13157) Follow-on improvements to hadoop credential commands
[
https://issues.apache.org/jira/browse/HADOOP-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Yoder updated HADOOP-13157:
Attachment: HADOOP-13157.004.branch-2.patch
Attaching patch 4 for branch-2. Sorry for the trouble.
> Follow-on improvements to hadoop credential commands
>
>
> Key: HADOOP-13157
> URL: https://issues.apache.org/jira/browse/HADOOP-13157
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0
>Reporter: Mike Yoder
>Assignee: Mike Yoder
> Fix For: 2.9.0
>
> Attachments: HADOOP-13157.001.patch, HADOOP-13157.002.patch,
> HADOOP-13157.003.branch-2.8.patch, HADOOP-13157.004.branch-2.patch
>
>
> [~andrew.wang] had some follow-up code review comments from HADOOP-12942.
> Hence this issue.
> Ping [~lmccay] as well.
> The comments:
> {quote}
> Overall this looks okay, the only correctness question I have is about the
> difference in behavior when the pwfile doesn't exist.
> The rest are all nits, would be nice to do these cleanups though.
> File
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java:
> Line 147:
> Could this be a static helper?
> Line 161: new
> The javadoc says it returns null in this situation. This is also a difference
> from the implementation in the AbstractJKSP. Intentional?
> Line 175: private void locateKeystore() throws IOException {
> static helper? for the construct*Path methods too?
> File
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyShell.java:
> Line 50: @VisibleForTesting public static final String NO_VALID_PROVIDERS =
> FYI for the future, our coding style is to put annotations on their own
> separate line.
> File
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java:
> Line 326: private char[] locatePassword() throws IOException {
> this method looks very similar to the one in JavaKeyStoreProvider, except the
> env var it looks for is different, is there potential for code reuse?
> Line 394: "o In the environment variable " +
> Using a "*" is the usual way of doing a bullet point, e.g. markdown and wiki
> syntax.
> Line 399:
> "http://hadoop.apache.org/docs/current/hadoop-project-dist/; +
> This link is not tied to a version, so could be inaccurate.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13010) Refactor raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289544#comment-15289544 ] Kai Zheng commented on HADOOP-13010: Hi Colin, Thanks for the comments. About the factories, I have to clarify the real problem in details and hope this works since the f2f discussion isn't going into details due to time constraint. We may have the following codecs in the 1st level: rs-legacy, rs-default (both belonging to RS) xor, hh or hitchhiker, lrc, ... And for each codec, it may use one or more raw coders, but each of such coders may use different implementations. For example, for the rs-default codec, we have two coder implementations (the pure java one and the isa-l one). Users may add their own coder implementation for a codec, maybe for better performance. So that's why I would have a configuration key like this: o.a.h.io.erasurecode.codec.(codec-name).rawcoder: (whatever value to be used to create or load the coder). Currently we configured the factory to create the encoder and decoder for a coder implementation, I agree there could be better option here, and while discussing about this in details with Andrew yesterday in the SF office, wonder if we could achieve the effect avoding the factories using java service loader. First, we can add codec-name and coder-name to the raw coder, so each coder will have a codec-name and coder-name when it's created. Then we have the built-in coders of fixed codec-name and coder-name. Customized coders will be loaded via service loader. Eventually we will have all the raw erasure coders loaded and created, then we can setup a mapping between codec-name and coder-name, coder-name and the coder-class or instance. Does this sound good to you? If it works, then we might do this in a follow-on task? Thanks again! > Refactor raw erasure coders > --- > > Key: HADOOP-13010 > URL: https://issues.apache.org/jira/browse/HADOOP-13010 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-13010-v1.patch, HADOOP-13010-v2.patch, > HADOOP-13010-v3.patch, HADOOP-13010-v4.patch, HADOOP-13010-v5.patch > > > This will refactor raw erasure coders according to some comments received so > far. > * As discussed in HADOOP-11540 and suggested by [~cmccabe], better not to > rely class inheritance to reuse the codes, instead they can be moved to some > utility. > * Suggested by [~jingzhao] somewhere quite some time ago, better to have a > state holder to keep some checking results for later reuse during an > encode/decode call. > This would not get rid of some inheritance levels as doing so isn't clear yet > for the moment and also incurs big impact. I do wish the end result by this > refactoring will make all the levels more clear and easier to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289541#comment-15289541
]
Hadoop QA commented on HADOOP-13171:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 46s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 18s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 21s
{color} | {color:red} root: The patch generated 20 new + 76 unchanged - 9 fixed
= 96 total (was 85) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 41s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 25s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black}
[jira] [Updated] (HADOOP-12767) update apache httpclient version to the latest 4.5 for security
[ https://issues.apache.org/jira/browse/HADOOP-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12767: Status: Patch Available (was: Open) > update apache httpclient version to the latest 4.5 for security > --- > > Key: HADOOP-12767 > URL: https://issues.apache.org/jira/browse/HADOOP-12767 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.7.2 >Reporter: Artem Aliev >Assignee: Artem Aliev > Attachments: HADOOP-12767-branch-2-005.patch, > HADOOP-12767-branch-2.004.patch, HADOOP-12767.001.patch, > HADOOP-12767.002.patch, HADOOP-12767.003.patch, HADOOP-12767.004.patch > > > Various SSL security fixes are needed. See: CVE-2012-6153, CVE-2011-4461, > CVE-2014-3577, CVE-2015-5262. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289510#comment-15289510
]
Hadoop QA commented on HADOOP-13171:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 3s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 17s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
8s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 24s
{color} | {color:red} root: The patch generated 26 new + 77 unchanged - 8 fixed
= 103 total (was 85) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-aws in the patch failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s
{color} | {color:red} hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101
generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 3s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 51s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not
[jira] [Commented] (HADOOP-13168) Support Future.get with timeout in ipc async calls
[ https://issues.apache.org/jira/browse/HADOOP-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289495#comment-15289495 ] Tsz Wo Nicholas Sze commented on HADOOP-13168: -- All of the checkstyle warnings are bogus, unfortunately. > Support Future.get with timeout in ipc async calls > -- > > Key: HADOOP-13168 > URL: https://issues.apache.org/jira/browse/HADOOP-13168 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: c13168_20160517.patch > > > Currently, the Future returned by ipc async call only support Future.get() > but not Future.get(timeout, unit). We should support the latter as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13138) Unable to append to a SequenceFile with Compression.NONE.
[
https://issues.apache.org/jira/browse/HADOOP-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289497#comment-15289497
]
Vinayakumar B commented on HADOOP-13138:
Thanks Chris
> Unable to append to a SequenceFile with Compression.NONE.
> -
>
> Key: HADOOP-13138
> URL: https://issues.apache.org/jira/browse/HADOOP-13138
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Gervais Mickaël
>Assignee: Vinayakumar B
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HADOOP-13138-01.patch, HADOOP-13138-02.patch
>
>
> Hi,
> I'm trying to use the append functionnality to an existing _SequenceFile_.
> If I set _Compression.NONE_, it works when the file is created, but when the
> file already exists I've a _NullPointerException_, by the way it works if I
> specify a compression with a codec.
> {code:title=Failing code|borderStyle=solid}
> Option compression = compression(CompressionType.NONE);
> Option keyClass = keyClass(LongWritable.class);
> Option valueClass = valueClass(BytesWritable.class);
> Option out = file(dfs);
> Option append = appendIfExists(true);
> writer = createWriter(conf,
> out,
> append,
> compression,
> keyClass,
> valueClass);
> {code}
> The following exeception is thrown when the file exists because compression
> option is checked:
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1119)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
> {code}
> This is due to the *codec* which is _null_:
> {code:title=SequenceFile.java|borderStyle=solid}
> if (readerCompressionOption.value != compressionTypeOption.value
> || !readerCompressionOption.codec.getClass().getName()
>
> .equals(compressionTypeOption.codec.getClass().getName())) {
> throw new IllegalArgumentException(
> "Compression option provided does not match the file");
> }
> {code}
> Thansk
> Mickaël
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[
https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289480#comment-15289480
]
Hadoop QA commented on HADOOP-13174:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-common-project: The patch generated 3 new + 142
unchanged - 0 fixed = 145 total (was 142) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 42s
{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 12s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804707/HADOOP-13174.01.patch
|
| JIRA Issue | HADOOP-13174 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b8683d079e1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ef17577 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9493/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9493/testReport/ |
| modules | C: hadoop-common-project/hadoop-auth
hadoop-common-project/hadoop-common U: hadoop-common-project |
| Console output |
[jira] [Commented] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289451#comment-15289451 ] Chris Douglas commented on HADOOP-13175: Really? Wow, neat. OK, feel free to close as WONTFIX. I just assumed it was a forgotten fragment. > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13008) Add XFS Filter for UIs to Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289443#comment-15289443 ] Chris Nauroth commented on HADOOP-13008: [~lmccay], the proposed refactoring sounds good to me. [~vvasudev], thank you for pointing out the duplication. > Add XFS Filter for UIs to Hadoop Common > --- > > Key: HADOOP-13008 > URL: https://issues.apache.org/jira/browse/HADOOP-13008 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 2.8.0 > > Attachments: HADOOP-13008-001.patch, HADOOP-13008-002.patch, > HADOOP-13008-003.patch, HADOOP-13008-004.patch > > > Cross Frame Scripting (XFS) prevention for UIs can be provided through a > common servlet filter. This filter will set the X-Frame-Options HTTP header > to DENY unless configured to another valid setting. > There are a number of UIs that could just add this to their filters as well > as the Yarn webapp proxy which could add it for all it's proxied UIs - if > appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12964) Http server vulnerable to clickjacking
[ https://issues.apache.org/jira/browse/HADOOP-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-12964: --- Fix Version/s: 2.9.0 > Http server vulnerable to clickjacking > --- > > Key: HADOOP-12964 > URL: https://issues.apache.org/jira/browse/HADOOP-12964 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0 > > Attachments: hadoop-12964.001.patch, hadoop-12964.002.patch, > hadoop-12964.003.patch > > > Nessus report shows a medium level issue that "Web Application Potentially > Vulnerable to Clickjacking" with the description as follows: > "The remote web server does not set an X-Frame-Options response header in all > content responses. This could potentially expose the site to a clickjacking > or UI Redress attack wherein an attacker can trick a user into clicking an > area of the vulnerable page that is different than what the user perceives > the page to be. This can result in a user performing fraudulent or malicious > transactions" > We could add X-Frame-Options, supported in all major browsers, in the Http > response header to mitigate the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289431#comment-15289431 ] Jason Lowe commented on HADOOP-13175: - Yes, we have internal users that requested the functionality be restored. I'll mention that it is likely to go away in Apache Hadoop in 3.x and see how much backlash I get. > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-12847:
-
Status: Patch Available (was: In Progress)
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch, HADOOP-12847.004.patch, HADOOP-12847.005.patch
>
>
> {{hadoop daemonlog}} is a simple, yet useful tool for debugging.
> However, it does not support https, nor does it support a Kerberized Hadoop
> cluster.
> Using {{AuthenticatedURL}}, it will be able to support SPNEGO negotiation
> with a Kerberized name node web ui. It will also fall back to simple
> authentication if the cluster is not Kerberized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13173) SecurityUtil.buildtokenService broke token authentication in multi-home network env
[ https://issues.apache.org/jira/browse/HADOOP-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-13173. Resolution: Duplicate Hello [~wuyuren]. I believe this issue is a duplicate of what was reported in HADOOP-12954, which has a patch committed for release 2.9.0, and MAPREDUCE-6565, which remains unresolved. I'm going to close this issue as a duplicate. > SecurityUtil.buildtokenService broke token authentication in multi-home > network env > --- > > Key: HADOOP-13173 > URL: https://issues.apache.org/jira/browse/HADOOP-13173 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 > Environment: multi-homed network cluster on SLES 11 >Reporter: Yuren Wu > > Hadoop cluster: multihomed network with KMS and Kerberos. > Sympton: mapreduce mapper will fail with kerberos error. (stack trace later) > Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET. > When mapreduce job submitted to the cluster from an outside ndoe (node only > has visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET > ip address. Such token in Credential's token list using IP will not be > correct resolved in the cluster. Therefore, we are using > hadoop.security.token.service.use_ip=false in the client side core-site.xml > file. > However, once job is accepted by yarn, the mapper container still throws > kerberos error. A detailed trace shows that when DFSClient is calling > decryptEncryptedDataencryptionKey function, it will reach to > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection > method. This method will call SecurityUtil.buildtokenService to obtain the > service name and use it to lookup the token in Credentials.getToken(service) > call. This call returned the service in the IP:PORT format. Therefore, no > token can be retrieved from credential's token hashmap. > After read SecurityUtil class, I think this class does not load configuration > from configuration file at all. It seems to me that this class create a > default configuration object and the useIpForTokenService variable is always > set default value true.. > I do not have enough experience to provide a fix. In my mind, somewhere we > need to provide the correct value of useIpForTokenService based on job setup. > Stack trace provided here. this is from wordcount job from example. > 2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at
[jira] [Updated] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-13175: --- Status: Patch Available (was: Open) > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-13175: --- Attachment: HADOOP-13175.001.patch > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13008) Add XFS Filter for UIs to Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289320#comment-15289320 ] Varun Vasudev edited comment on HADOOP-13008 at 5/18/16 5:37 PM: - I prefer the generic XFS filter based approach to the QuotingInputFilter - it's more flexible. The only reason I realized this is that I was testing a patch for YARN integration and noticed that the header was being set for all responses and I wasn't sure why. I defer to Chris and you on what to do going forward. was (Author: vvasudev): I prefer the filter based approach - it's more flexible. The only reason I realized this is that I was testing a patch for YARN integration and noticed that the header was being set for all responses and I wasn't sure why. I defer to Chris and you on what to do going forward. > Add XFS Filter for UIs to Hadoop Common > --- > > Key: HADOOP-13008 > URL: https://issues.apache.org/jira/browse/HADOOP-13008 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 2.8.0 > > Attachments: HADOOP-13008-001.patch, HADOOP-13008-002.patch, > HADOOP-13008-003.patch, HADOOP-13008-004.patch > > > Cross Frame Scripting (XFS) prevention for UIs can be provided through a > common servlet filter. This filter will set the X-Frame-Options HTTP header > to DENY unless configured to another valid setting. > There are a number of UIs that could just add this to their filters as well > as the Yarn webapp proxy which could add it for all it's proxied UIs - if > appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12767) update apache httpclient version to the latest 4.5 for security
[ https://issues.apache.org/jira/browse/HADOOP-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12767: Attachment: HADOOP-12767-branch-2-005.patch HADOOP-12767 patch 005: fix up checkstyle, move to httpclient v. 4.5.2 —the latest. checkstyle will complain about method length; javac about deprecation. None of those can be helped. > update apache httpclient version to the latest 4.5 for security > --- > > Key: HADOOP-12767 > URL: https://issues.apache.org/jira/browse/HADOOP-12767 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.7.2 >Reporter: Artem Aliev >Assignee: Artem Aliev > Attachments: HADOOP-12767-branch-2-005.patch, > HADOOP-12767-branch-2.004.patch, HADOOP-12767.001.patch, > HADOOP-12767.002.patch, HADOOP-12767.003.patch, HADOOP-12767.004.patch > > > Various SSL security fixes are needed. See: CVE-2012-6153, CVE-2011-4461, > CVE-2014-3577, CVE-2015-5262. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12767) update apache httpclient version to the latest 4.5 for security
[ https://issues.apache.org/jira/browse/HADOOP-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289396#comment-15289396 ] Steve Loughran commented on HADOOP-12767: - tested all the openstack; all is happy. I'll have a gol with the changes here, and bump up to the same artifact of httpclient as HADOOP-13044 proposed > update apache httpclient version to the latest 4.5 for security > --- > > Key: HADOOP-12767 > URL: https://issues.apache.org/jira/browse/HADOOP-12767 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.7.2 >Reporter: Artem Aliev >Assignee: Artem Aliev > Attachments: HADOOP-12767-branch-2.004.patch, HADOOP-12767.001.patch, > HADOOP-12767.002.patch, HADOOP-12767.003.patch, HADOOP-12767.004.patch > > > Various SSL security fixes are needed. See: CVE-2012-6153, CVE-2011-4461, > CVE-2014-3577, CVE-2015-5262. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12767) update apache httpclient version to the latest 4.5 for security
[ https://issues.apache.org/jira/browse/HADOOP-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12767: Status: Open (was: Patch Available) > update apache httpclient version to the latest 4.5 for security > --- > > Key: HADOOP-12767 > URL: https://issues.apache.org/jira/browse/HADOOP-12767 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.7.2 >Reporter: Artem Aliev >Assignee: Artem Aliev > Attachments: HADOOP-12767-branch-2.004.patch, HADOOP-12767.001.patch, > HADOOP-12767.002.patch, HADOOP-12767.003.patch, HADOOP-12767.004.patch > > > Various SSL security fixes are needed. See: CVE-2012-6153, CVE-2011-4461, > CVE-2014-3577, CVE-2015-5262. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12782) Faster LDAP group name resolution with ActiveDirectory
[ https://issues.apache.org/jira/browse/HADOOP-12782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289397#comment-15289397 ] Wei-Chiu Chuang commented on HADOOP-12782: -- The failed test is a know flaky test, unrelated to this patch. > Faster LDAP group name resolution with ActiveDirectory > -- > > Key: HADOOP-12782 > URL: https://issues.apache.org/jira/browse/HADOOP-12782 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HADOOP-12782.001.patch, HADOOP-12782.002.patch, > HADOOP-12782.003.patch, HADOOP-12782.004.patch, HADOOP-12782.005.patch, > HADOOP-12782.006.patch, HADOOP-12782.007.patch, HADOOP-12782.008.patch, > HADOOP-12782.009.patch > > > The typical LDAP group name resolution works well under typical scenarios. > However, we have seen cases where a user is mapped to many groups (in an > extreme case, a user is mapped to more than 100 groups). The way it's being > implemented now makes this case super slow resolving groups from > ActiveDirectory. > The current LDAP group resolution implementation sends two queries to a > ActiveDirectory server. The first query returns a user object, which contains > DN (distinguished name). The second query looks for groups where the user DN > is a member. If a user is mapped to many groups, the second query returns all > group objects associated with the user, and is thus very slow. > After studying a user object in ActiveDirectory, I found a user object > actually contains a "memberOf" field, which is the DN of all group objects > where the user belongs to. Assuming that an organization has no recursive > group relation (that is, a user A is a member of group G1, and group G1 is a > member of group G2), we can use this properties to avoid the second query, > which can potentially run very slow. > I propose that we add a configuration to only enable this feature for users > who want to reduce group resolution time and who does not have recursive > groups, so that existing behavior will not be broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-002.patch
Patch 002; identify and fix up some checkstyle and javadoc issues before Yetus
does
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13174: --- Target Version/s: 2.8.0 Status: Patch Available (was: Open) > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Attachments: HADOOP-13174.01.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289377#comment-15289377 ] Xiao Chen commented on HADOOP-13174: Attached initial patch. [~yzhangal], could you please take a look? Thanks! > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Attachments: HADOOP-13174.01.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289375#comment-15289375 ] Chris Douglas commented on HADOOP-13175: [~jlowe], you restored this in HADOOP-11007. Does this still have users you're aware of? > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
Chris Douglas created HADOOP-13175: -- Summary: Remove hadoop-ant from hadoop-tools Key: HADOOP-13175 URL: https://issues.apache.org/jira/browse/HADOOP-13175 Project: Hadoop Common Issue Type: Task Reporter: Chris Douglas The hadoop-ant code is an ancient kludge unlikely to have any users, still. We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13174: --- Description: Recently I debugged several authentication related problems, and found that the debug logs are not enough to identify a problem. This jira improves it by adding more debug/trace logs along the line. > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Attachments: HADOOP-13174.01.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13174: --- Attachment: HADOOP-13174.01.patch > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Attachments: HADOOP-13174.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
Xiao Chen created HADOOP-13174: -- Summary: Add more debug logs for delegation tokens and authentication Key: HADOOP-13174 URL: https://issues.apache.org/jira/browse/HADOOP-13174 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Xiao Chen Assignee: Xiao Chen Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12782) Faster LDAP group name resolution with ActiveDirectory
[
https://issues.apache.org/jira/browse/HADOOP-12782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289371#comment-15289371
]
Hadoop QA commented on HADOOP-12782:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 40 unchanged - 7 fixed = 40 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 55s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804693/HADOOP-12782.009.patch
|
| JIRA Issue | HADOOP-12782 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 43d0bc81fe77 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f4d8fde |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9490/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9490/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9490/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9490/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Faster LDAP
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13010) Refactor raw erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289361#comment-15289361
]
Colin Patrick McCabe commented on HADOOP-13010:
---
bq. Not sure if it's good to have something like isXOR or isRS, because we'll
have more coder algorithms other than the current both.
That's a fair point. It seems unlikely that we need an isXOR or isRS method.
bq. OK, we can have \[createRawEncoder\]. Maybe it wouldn't be bad to have two
shortcut methods additionally, createRSRawEncoder and createXORRawEncoder,
because the both are the primitive, essential and most used ones in
implementing advanced coders and HDFS side. I want the both to be outstanding
and easily used.
It seems better just to have one function, {{createRawEncoder}}, than to have
lots of functions for every type of encoder.
bq. I discussed about this with Uma Maheswara Rao G quite some ago when
introducing these factories. There isn't a clear way to compose or reduce the
full class name of a raw coder because it should be plugin-able and
configurable. In current approach, for each codec, there could be some coder
implementations, and for each, the corresponding coder factory can be
configured.
We discussed this offline and I think the conclusion is that we don't need the
factories for anything.
We can just have a configuration key like {{erasure.coder.algorithm}} and then
code like this:
{code}
RawErasureEncoder createRawEncoder(Configuration conf) {
String classPrefix = conf.get("erasure.coder.algorithm",
DEFAULT_ERASURE_CODER_ALGORITHM);
String name = classPrefix + "Encoder";
Constructor ctor =
classLoader.loadClass(name).getConstructor(Configuration.class);
return ctor.newInstance(conf);
}
RawErasureDecoder createRawDecoder(Configuration conf) {
String classPrefix = conf.get("erasure.coder.algorithm",
DEFAULT_ERASURE_CODER_ALGORITHM);
String name = classPrefix + "Decoder";
Constructor ctor =
classLoader.loadClass(name).getConstructor(Configuration.class);
return ctor.newInstance(conf);
}
{code}
bq. It seems this can simplify the related functions, but am not sure it would
make the codes more readable. The mentioned variables are very specific to
encode/decode related calls using on-heap bytebuffer or byte array buffers.
Maybe DecodingState could be kept simple not putting too many intermediate
variables because the codes using of them are not suitable to be moved to the
class.
Reducing the number of function parameters from 8 or 9 to 1 or 2 seems like it
would make the code much more readable. I don't understand what the rationale
is for keeping these parameters out of DecodingState. Perhaps we could discuss
this in a follow-on JIRA, though.
> Refactor raw erasure coders
> ---
>
> Key: HADOOP-13010
> URL: https://issues.apache.org/jira/browse/HADOOP-13010
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-13010-v1.patch, HADOOP-13010-v2.patch,
> HADOOP-13010-v3.patch, HADOOP-13010-v4.patch, HADOOP-13010-v5.patch
>
>
> This will refactor raw erasure coders according to some comments received so
> far.
> * As discussed in HADOOP-11540 and suggested by [~cmccabe], better not to
> rely class inheritance to reuse the codes, instead they can be moved to some
> utility.
> * Suggested by [~jingzhao] somewhere quite some time ago, better to have a
> state holder to keep some checking results for later reuse during an
> encode/decode call.
> This would not get rid of some inheritance levels as doing so isn't clear yet
> for the moment and also incurs big impact. I do wish the end result by this
> refactoring will make all the levels more clear and easier to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
