[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.47
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695097#comment-15695097 ] Akira Ajisaka commented on HADOOP-13812: Thank you for the quick response! How about upgrading to 6.0.48? This version includes some security fixes. https://tomcat.apache.org/security-6.html#Fixed_in_Apache_Tomcat_6.0.48 > Upgrade Tomcat to 6.0.47 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.47. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13820) [viewfs] Listfile gives complete Realm as User
[
https://issues.apache.org/jira/browse/HADOOP-13820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695088#comment-15695088
]
Akira Ajisaka commented on HADOOP-13820:
Mostly looks good to me. Hi [~brahmareddy], would you rebase the patch and fix
the checkstyle warning?
> [viewfs] Listfile gives complete Realm as User
> --
>
> Key: HADOOP-13820
> URL: https://issues.apache.org/jira/browse/HADOOP-13820
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Archana T
>Assignee: Brahma Reddy Battula
>Priority: Minor
> Attachments: HADOOP-13820-002.patch, HADOOP-13820-003.patch,
> HADOOP-13820-004.patch, HADOOP-13820.patch
>
>
> When defaultFS is configured as viewfs --
> fs.defaultFS
> viewfs://CLUSTER/
> List Files showing Realm as User --
> hdfs dfs -ls /
> Found 2 items
> -r-xr-xr-x - {color:red} h...@hadoop.com {color} hadoop 0
> 2016-11-07 15:31 /Dir1
> -r-xr-xr-x - {color:red} h...@hadoop.com {color} hadoop 0
> 2016-11-07 15:31 /Dir2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.47
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695085#comment-15695085 ] John Zhuge commented on HADOOP-13812: - Tested 6.0.47 with CDH 5.10 in the following cases: * KMS, kerberos * KMS, kerberos + SSL * HttpFS, kerberos Investigating some issue in this case: * HttpFS, kerberos + SSL > Upgrade Tomcat to 6.0.47 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.47. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.47
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695034#comment-15695034 ] Akira Ajisaka commented on HADOOP-13812: Hi [~jzhuge], how is this issue going on? > Upgrade Tomcat to 6.0.47 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.47. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12549) Extend HDFS-7456 default generically to all pattern lookups
[
https://issues.apache.org/jira/browse/HADOOP-12549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694990#comment-15694990
]
Harsh J commented on HADOOP-12549:
--
[~daryn] / [~yzhangal] - Could you help take a look at this one?
> Extend HDFS-7456 default generically to all pattern lookups
> ---
>
> Key: HADOOP-12549
> URL: https://issues.apache.org/jira/browse/HADOOP-12549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, security
>Affects Versions: 2.7.1
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
> Attachments: HADOOP-12549.patch
>
>
> In HDFS-7546 we added a hdfs-default.xml property to bring back the regular
> behaviour of trusting all principals (as was the case before HADOOP-9789).
> However, the change only targeted HDFS users and also only those that used
> the default-loading mechanism of Configuration class (i.e. not {{new
> Configuration(false)}} users).
> I'd like to propose adding the same default to the generic RPC client code
> also, so the default affects all form of clients equally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694987#comment-15694987 ] Harsh J commented on HADOOP-6801: - [~cnauroth] / [~ajisakaa] - Could you help take a look at this one? > io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are > still in CommonConfigurationKeysPublic.java and used in SequenceFile.java > --- > > Key: HADOOP-6801 > URL: https://issues.apache.org/jira/browse/HADOOP-6801 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Erik Steffl >Assignee: Harsh J >Priority: Minor > Attachments: HADOOP-6801.05.patch, HADOOP-6801.r1.diff, > HADOOP-6801.r2.diff, HADOOP-6801.r3.diff, HADOOP-6801.r4.diff, > HADOOP-6801.r5.diff > > > Following configuration keys in CommonConfigurationKeysPublic.java (former > CommonConfigurationKeys.java): > public static final String IO_SORT_MB_KEY = "io.sort.mb"; > public static final String IO_SORT_FACTOR_KEY = "io.sort.factor"; > are partially moved: > - they were renamed to mapreduce.task.io.sort.mb and > mapreduce.task.io.sort.factor respectively > - they were moved to mapreduce project, documented in mapred-default.xml > However: > - they are still listed in CommonConfigurationKeysPublic.java as quoted > above > - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java > in Hadoop Common project > Not sure what the solution is, these constants should probably be removed > from CommonConfigurationKeysPublic.java but I am not sure what's the best > solution for SequenceFile.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9424) The "hadoop jar" invocation should include the passed jar on the classpath as a whole
[
https://issues.apache.org/jira/browse/HADOOP-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HADOOP-9424:
Assignee: (was: Harsh J)
> The "hadoop jar" invocation should include the passed jar on the classpath as
> a whole
> -
>
> Key: HADOOP-9424
> URL: https://issues.apache.org/jira/browse/HADOOP-9424
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Priority: Minor
> Attachments: HADOOP-9424.patch
>
>
> When you have a case such as this:
> {{X.jar -> Classes = Main, Foo}}
> {{Y.jar -> Classes = Bar}}
> With implementation details such as:
> * Main references Bar and invokes a public, static method on it.
> * Bar does a class lookup to find Foo (Class.forName("Foo")).
> Then when you do a {{HADOOP_CLASSPATH=Y.jar hadoop jar X.jar Main}}, the
> Bar's method fails with a ClassNotFound exception cause of the way RunJar
> runs.
> RunJar extracts the passed jar and includes its contents on the ClassLoader
> of its current thread but the {{Class.forName(…)}} call from another class
> does not check that class loader and hence cannot find the class as its not
> on any classpath it is aware of.
> The script of "hadoop jar" should ideally include the passed jar argument to
> the CLASSPATH before RunJar is invoked, for this above case to pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694982#comment-15694982
]
Harsh J commented on HADOOP-13817:
--
[~xyao] - Thanks for the reviews so far. Could you take a look at the current
patch-set?
[~yzhangal] - Could you help take a look at this one?
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-1381) The distance between sync blocks in SequenceFiles should be configurable
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694979#comment-15694979 ] Harsh J commented on HADOOP-1381: - [~ajisakaa] / [~ozawa]] - Could you help take a look at this one? > The distance between sync blocks in SequenceFiles should be configurable > > > Key: HADOOP-1381 > URL: https://issues.apache.org/jira/browse/HADOOP-1381 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.0.0-alpha >Reporter: Owen O'Malley >Assignee: Harsh J > Attachments: HADOOP-1381.r1.diff, HADOOP-1381.r2.diff, > HADOOP-1381.r3.diff, HADOOP-1381.r4.diff, HADOOP-1381.r5.diff, > HADOOP-1381.r5.diff > > > Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much > better if it was configurable with a much higher default (1mb or so?). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13694) Data transfer encryption with AES 192: Invalid key length.
[ https://issues.apache.org/jira/browse/HADOOP-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694976#comment-15694976 ] Harsh J commented on HADOOP-13694: -- [~cnauroth] / [~hitliuyi] - Would you be able to take a look at this one? > Data transfer encryption with AES 192: Invalid key length. > -- > > Key: HADOOP-13694 > URL: https://issues.apache.org/jira/browse/HADOOP-13694 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.7.2 > Environment: OS: Ubuntu 14.04 > /hadoop-2.7.2/bin$ uname -a > Linux wkstn-kpalaniappan 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 > 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > /hadoop-2.7.2/bin$ java -version > java version "1.7.0_95" > OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1) > OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) > Hadoop version: 2.7.2 >Reporter: Karthik Palaniappan >Assignee: Harsh J > > Configuring aes 128 or aes 256 encryption > (dfs.encrypt.data.transfer.cipher.key.bitlength = [128, 256]) works perfectly > fine. Trying to use AES 192 generates this exception on the datanode: > 16/02/29 17:34:10 ERROR datanode.DataNode: > wkstn-kpalaniappan:50010:DataXceiver error processing unknown operation src: > /127.0.0.1:57237 dst: /127.0.0.1:50010 > java.lang.IllegalArgumentException: Invalid key length. > at org.apache.hadoop.crypto.OpensslCipher.init(Native Method) > at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176) > at > org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116) > at > org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290) > at > org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:396) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193) > at java.lang.Thread.run(Thread.java:745) > And this exception on the client: > /hadoop-2.7.2/bin$ ./hdfs dfs -copyFromLocal ~/.vimrc /vimrc > 16/02/29 17:34:10 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Invalid key length. > at org.apache.hadoop.crypto.OpensslCipher.init(Native Method) > at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176) > at > org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116) > at > org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290) > at > org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:490) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:
[jira] [Commented] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694698#comment-15694698
]
Brahma Reddy Battula commented on HADOOP-13833:
---
[~ste...@apache.org] thanks for review and commit.
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13831) Correct check for error code to detect Azure Storage Throttling and provide retries
[ https://issues.apache.org/jira/browse/HADOOP-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694330#comment-15694330 ] Gaurav Kanade commented on HADOOP-13831: yes tagged the component; not sure I understand exactly what is meant by which version this 'surfaces' on - is it affects version or fix version ? > Correct check for error code to detect Azure Storage Throttling and provide > retries > --- > > Key: HADOOP-13831 > URL: https://issues.apache.org/jira/browse/HADOOP-13831 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Gaurav Kanade > > Azure Storage throttling affects HBase operations such as archiving old > WALS and others. In such cases the storage driver needs to detect and handle > the exception. We put in this logic to do the retries however the condition > to check for the exception is not always met due to inconsistency in which > the manner the error code is passed back. Instead the retry logic should > check for http status code (503) which is more reliable and consistent check -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13831) Correct check for error code to detect Azure Storage Throttling and provide retries
[ https://issues.apache.org/jira/browse/HADOOP-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Kanade updated HADOOP-13831: --- Component/s: fs/azure > Correct check for error code to detect Azure Storage Throttling and provide > retries > --- > > Key: HADOOP-13831 > URL: https://issues.apache.org/jira/browse/HADOOP-13831 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Gaurav Kanade > > Azure Storage throttling affects HBase operations such as archiving old > WALS and others. In such cases the storage driver needs to detect and handle > the exception. We put in this logic to do the retries however the condition > to check for the exception is not always met due to inconsistency in which > the manner the error code is passed back. Instead the retry logic should > check for http status code (503) which is more reliable and consistent check -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694032#comment-15694032
]
Hadoop QA commented on HADOOP-12804:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-12804 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840434/HADOOP-12804-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d9d42df22da6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 01665e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11136/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11136/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Read Proxy Password from Credential Providers in S3 FileSystem
> --
>
> Key: HADOOP-12804
> URL: https://issues.apache.org/jira/browse/HADOOP-12804
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Larry McCay
>Assignee: Larry McCay
>Priority: Minor
> Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch,
> HADOOP-12804-004.patch, HADOOP-12804-005.patch, HADOOP-12804-006.patch
[jira] [Comment Edited] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694003#comment-15694003 ] Larry McCay edited comment on HADOOP-12804 at 11/24/16 6:53 PM: Patch v006 addresses [~ste...@apache.org]'s review comments. Testing has been done against S3 bucket in US East (N. Virginia) running on EC2. was (Author: lmccay): Patch v005 addresses [~ste...@apache.org]'s review comments. Testing has been done against S3 bucket in US East (N. Virginia) running on EC2. > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-004.patch, HADOOP-12804-005.patch, HADOOP-12804-006.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694003#comment-15694003 ] Larry McCay commented on HADOOP-12804: -- Patch v005 addresses [~ste...@apache.org]'s review comments. Testing has been done against S3 bucket in US East (N. Virginia) running on EC2. > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-004.patch, HADOOP-12804-005.patch, HADOOP-12804-006.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-12804: - Status: Patch Available (was: Open) > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-alpha1, 2.8.0 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-004.patch, HADOOP-12804-005.patch, HADOOP-12804-006.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693961#comment-15693961 ] Hudson commented on HADOOP-10776: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10890 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10890/]) HADOOP-10776 Open up already widely-used APIs for delegation-token (stevel: rev 01665e456de8d79000ce273dded5ea53aa62965a) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AccessControlException.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AuthorizationException.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch, HADOOP-10776-branch-2-003.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693927#comment-15693927
]
Hudson commented on HADOOP-13833:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10889 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10889/])
HADOOP-13833 TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2
(stevel: rev 3df3fa396c7b338928f77344edd8c4dcda957671)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch, HADOOP-10776-branch-2-003.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693919#comment-15693919
]
Hadoop QA commented on HADOOP-13823:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
37s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-13823 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840444/HADOOP-13823-branch-2-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 0c26c8df0a02 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 1edabc4 |
| Default Java | 1.7.0_121 |
[jira] [Commented] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693902#comment-15693902 ] Steve Loughran commented on HADOOP-10776: - +1 to Vinod's patch with my minor changes; committing to branch-2.8+ > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch, HADOOP-10776-branch-2-003.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13833:
Resolution: Fixed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
thanks for fixing this so fast.
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13833:
Affects Version/s: 2.9.0
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693883#comment-15693883
]
Steve Loughran commented on HADOOP-13833:
-
+1, will commit. Thanks for fixing this
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[
https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693885#comment-15693885
]
Hadoop QA commented on HADOOP-10776:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
14s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
43s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 266 unchanged - 5 fixed = 268 total (was 271) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
58s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-10776 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840439/HADOOP-10776-branch-2-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 91056b5db6aa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality
[jira] [Updated] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13833:
Priority: Critical (was: Major)
Description:
{noformat}
org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
was:<...ileSystem for scheme[ "null"]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
*REF:*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
was:
{noformat}
org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
was:<...ileSystem for scheme[ "null"]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
*REF:*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files
[
https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693855#comment-15693855
]
Hadoop QA commented on HADOOP-13018:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
33s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13018 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12802265/HADOOP-13018.04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a0f5dc39f47f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / eb0a483 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11132/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11132/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Make Kdiag check whether hadoop.token.files points to existent and valid files
> --
>
> Key: HADOOP-13018
> URL: https://issues.apache.org/jira/browse/HADOOP-13018
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Fix For: 2.9.0
>
> Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch,
> HADOOP-13018.03.patch, HADOOP-13018.04.pat
[jira] [Updated] (HADOOP-13834) use JUnit categories for s3a scale tests and others
[ https://issues.apache.org/jira/browse/HADOOP-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13834: Summary: use JUnit categories for s3a scale tests and others (was: use categories for s3a scale tests and others) > use JUnit categories for s3a scale tests and others > --- > > Key: HADOOP-13834 > URL: https://issues.apache.org/jira/browse/HADOOP-13834 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > I did the -Pscale thing in the wrong way. What I should have done is added a > junit4 category instead. This is cleaner, more elegant, and more flexible in > future (example, we could make the s3guard tests their own category) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13834) use categories for s3a scale tests and others
[ https://issues.apache.org/jira/browse/HADOOP-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13834: Summary: use categories for s3a scale tests and others (was: use categories for s3a scale) > use categories for s3a scale tests and others > - > > Key: HADOOP-13834 > URL: https://issues.apache.org/jira/browse/HADOOP-13834 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > I did the -Pscale thing in the wrong way. What I should have done is added a > junit4 category instead. This is cleaner, more elegant, and more flexible in > future (example, we could make the s3guard tests their own category) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13834) use categories for s3a scale
Steve Loughran created HADOOP-13834: --- Summary: use categories for s3a scale Key: HADOOP-13834 URL: https://issues.apache.org/jira/browse/HADOOP-13834 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 2.8.0 Reporter: Steve Loughran Assignee: Steve Loughran I did the -Pscale thing in the wrong way. What I should have done is added a junit4 category instead. This is cleaner, more elegant, and more flexible in future (example, we could make the s3guard tests their own category) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Status: Patch Available (was: Open)
rested: s3 ireland
> s3a rename: fail if dest file exists
>
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch,
> HADOOP-13823-branch-2-002.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693813#comment-15693813
]
Hadoop QA commented on HADOOP-13833:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13833 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840438/HADOOP-13833.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 20373b8012e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / eb0a483 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11131/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11131/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files
[ https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693810#comment-15693810 ] Hudson commented on HADOOP-13018: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10888 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10888/]) HADOOP-13018. Make Kdiag check whether hadoop.token.files points to (stevel: rev abb9fa7fc69ec7b25f1c44e17c4c7dd17f5de16a) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/KDiag.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestKDiagNoKDC.java > Make Kdiag check whether hadoop.token.files points to existent and valid files > -- > > Key: HADOOP-13018 > URL: https://issues.apache.org/jira/browse/HADOOP-13018 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 2.9.0 > > Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch, > HADOOP-13018.03.patch, HADOOP-13018.04.patch > > > Steve proposed that KDiag should fail fast to help debug the case that > hadoop.token.files points to a file not found. This JIRA is to affect that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11603) Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized
[
https://issues.apache.org/jira/browse/HADOOP-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693811#comment-15693811
]
Hudson commented on HADOOP-11603:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10888 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10888/])
HADOOP-11603. Metric Snapshot log can be changed #MetricsSystemImpl.java
(brahma: rev 13501053dd95f41ac14091f5bbd79bcb2a3896a3)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
> Metric Snapshot log can be changed #MetricsSystemImpl.java since all the
> services will be initialized
> -
>
> Key: HADOOP-11603
> URL: https://issues.apache.org/jira/browse/HADOOP-11603
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Minor
> Labels: BB2015-05-TBR
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-11603-002.patch, HADOOP-11603.patch,
> defaultmetricIntialize-classes.JPG
>
>
> Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice
> etc... will initialize DefaultMetricsSystem hence following log will be
> logged for all the classes.. Since there is snapshot feature in HDFS ,it's
> better to change the following...
> {code}
> LOG.info("Scheduled snapshot period at "+ period +" second(s).");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Attachment: HADOOP-13823-branch-2-002.patch
Patch 2; patch 1 with the parameter to a method with a changed name.
> s3a rename: fail if dest file exists
>
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch,
> HADOOP-13823-branch-2-002.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Status: Open (was: Patch Available)
> s3a rename: fail if dest file exists
>
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693792#comment-15693792
]
Steve Loughran commented on HADOOP-13823:
-
checkstyle is being fussy about param names, but may as well fix that
> s3a rename: fail if dest file exists
>
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11603) Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized
[
https://issues.apache.org/jira/browse/HADOOP-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-11603:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
[~ajisakaa] thanks a lot for review..Jenkins report is fine and it's just log
change hence test is not added.
Committed to trunk,branch-2 and branch-2.8.
> Metric Snapshot log can be changed #MetricsSystemImpl.java since all the
> services will be initialized
> -
>
> Key: HADOOP-11603
> URL: https://issues.apache.org/jira/browse/HADOOP-11603
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Minor
> Labels: BB2015-05-TBR
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-11603-002.patch, HADOOP-11603.patch,
> defaultmetricIntialize-classes.JPG
>
>
> Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice
> etc... will initialize DefaultMetricsSystem hence following log will be
> logged for all the classes.. Since there is snapshot feature in HDFS ,it's
> better to change the following...
> {code}
> LOG.info("Scheduled snapshot period at "+ period +" second(s).");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693754#comment-15693754
]
Hadoop QA commented on HADOOP-13823:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1
new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-13823 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840436/HADOOP-13813-branch-2-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 3497edcfbf3d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/pers
[jira] [Comment Edited] (HADOOP-13605) Clean up FileSystem javadocs, logging; improve diagnostics on FS load
[
https://issues.apache.org/jira/browse/HADOOP-13605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693617#comment-15693617
]
Brahma Reddy Battula edited comment on HADOOP-13605 at 11/24/16 4:48 PM:
-
Following test fails after this check-in , Raised HADOOP-13833 to track.
Pre-commit did not catch because it's from HDFS project
{{TestSymlinkHdfsFileSystem}}.
{noformat}
org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
was:<...ileSystem for scheme[ "null"]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
*REF:*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
was (Author: brahmareddy):
Following test fails after this check-in , Raised HADOOP-13833 to track.
{noformat}
org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
was:<...ileSystem for scheme[ "null"]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
*REF:*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
> Clean up FileSystem javadocs, logging; improve diagnostics on FS load
> -
>
> Key: HADOOP-13605
> URL: https://issues.apache.org/jira/browse/HADOOP-13605
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13605-004.patch, HADOOP-13605-005.patch,
> HADOOP-13605-branch-2-001.patch, HADOOP-13605-branch-2-002.patch,
> HADOOP-13605-branch-2-003.patch
>
>
> We can't easily debug FS instantiation problems as there isn't much detail in
> what was going on.
> We can add more logging, but cannot simply switch {{FileSystem.LOG}} to SLF4J
> —the class is used in too many places, including tests which cast it.
> Instead, add a new private SLF4J Logger, {{LOGGER}} and switch logging to it.
> While working in the base FileSystem class, take the opportunity to clean up
> javadocs and comments
> # add the list of exceptions, including indicating which base classes throw
> UnsupportedOperationExceptions
> # cut bits in the comments which are not true
> The outcome of this patch is that IDEs shouldn't highlight most of the file
> as flawed in some way or another
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files
[ https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13018: Resolution: Fixed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Applied to branch-2; Apologies; I'd missed this patch completely. Feel free to ping me if you think I'm neglecting something > Make Kdiag check whether hadoop.token.files points to existent and valid files > -- > > Key: HADOOP-13018 > URL: https://issues.apache.org/jira/browse/HADOOP-13018 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 2.9.0 > > Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch, > HADOOP-13018.03.patch, HADOOP-13018.04.patch > > > Steve proposed that KDiag should fail fast to help debug the case that > hadoop.token.files points to a file not found. This JIRA is to affect that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files
[ https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693708#comment-15693708 ] Steve Loughran commented on HADOOP-13018: - LTGM +1 > Make Kdiag check whether hadoop.token.files points to existent and valid files > -- > > Key: HADOOP-13018 > URL: https://issues.apache.org/jira/browse/HADOOP-13018 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch, > HADOOP-13018.03.patch, HADOOP-13018.04.patch > > > Steve proposed that KDiag should fail fast to help debug the case that > hadoop.token.files points to a file not found. This JIRA is to affect that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Status: Patch Available (was: Open) > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch, HADOOP-10776-branch-2-003.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Status: Patch Available (was: Open)
testing: s3a ireland
> s3a rename: fail if dest file exists
>
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Attachment: HADOOP-10776-branch-2-003.patch patch 003 address some of the checkstyle/javadoc warnings. > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch, HADOOP-10776-branch-2-003.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Status: Open (was: Patch Available) > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-13833:
--
Attachment: HADOOP-13833.patch
Uploading patch to fix the testcase..
[~ste...@apache.org]/[~liuml07] can you review the patch as you both worked on
HADOOP-13605..?
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
[
https://issues.apache.org/jira/browse/HADOOP-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-13833:
--
Assignee: Brahma Reddy Battula
Status: Patch Available (was: Open)
> TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
> HADOOP13605
> --
>
> Key: HADOOP-13833
> URL: https://issues.apache.org/jira/browse/HADOOP-13833
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-13833.patch
>
>
> {noformat}
> org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
> was:<...ileSystem for scheme[ "null"]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *REF:*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Summary: s3a rename: fail if dest file exists (was: s3a rename: fail if
dest file exists; throw FNFE if source is missing)
> s3a rename: fail if dest file exists
>
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists; throw FNFE if source is missing
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Attachment: HADOOP-13813-branch-2-001.patch
Patch 001. Modified behaviour; XML declaration and removed subclass of contract
test
> s3a rename: fail if dest file exists; throw FNFE if source is missing
> -
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-13813-branch-2-001.patch
>
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-12804: - Attachment: HADOOP-12804-006.patch > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-004.patch, HADOOP-12804-005.patch, HADOOP-12804-006.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-12804: - Status: Open (was: Patch Available) > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-alpha1, 2.8.0 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-004.patch, HADOOP-12804-005.patch, HADOOP-12804-006.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13605) Clean up FileSystem javadocs, logging; improve diagnostics on FS load
[
https://issues.apache.org/jira/browse/HADOOP-13605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693617#comment-15693617
]
Brahma Reddy Battula commented on HADOOP-13605:
---
Following test fails after this check-in , Raised HADOOP-13833 to track.
{noformat}
org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
was:<...ileSystem for scheme[ "null"]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
*REF:*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
> Clean up FileSystem javadocs, logging; improve diagnostics on FS load
> -
>
> Key: HADOOP-13605
> URL: https://issues.apache.org/jira/browse/HADOOP-13605
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13605-004.patch, HADOOP-13605-005.patch,
> HADOOP-13605-branch-2-001.patch, HADOOP-13605-branch-2-002.patch,
> HADOOP-13605-branch-2-003.patch
>
>
> We can't easily debug FS instantiation problems as there isn't much detail in
> what was going on.
> We can add more logging, but cannot simply switch {{FileSystem.LOG}} to SLF4J
> —the class is used in too many places, including tests which cast it.
> Instead, add a new private SLF4J Logger, {{LOGGER}} and switch logging to it.
> While working in the base FileSystem class, take the opportunity to clean up
> javadocs and comments
> # add the list of exceptions, including indicating which base classes throw
> UnsupportedOperationExceptions
> # cut bits in the comments which are not true
> The outcome of this patch is that IDEs shouldn't highlight most of the file
> as flawed in some way or another
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13833) TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP13605
Brahma Reddy Battula created HADOOP-13833:
-
Summary:
TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after
HADOOP13605
Key: HADOOP-13833
URL: https://issues.apache.org/jira/browse/HADOOP-13833
Project: Hadoop Common
Issue Type: Bug
Components: test
Reporter: Brahma Reddy Battula
{noformat}
org.junit.ComparisonFailure: expected:<...ileSystem for scheme[: null]> but
was:<...ileSystem for scheme[ "null"]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.fs.SymlinkBaseTest.testCreateLinkUsingPartQualPath2(SymlinkBaseTest.java:574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
*REF:*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/235/testReport/junit/org.apache.hadoop.fs/TestSymlinkHdfsFileSystem/testCreateLinkUsingPartQualPath2/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[
https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693551#comment-15693551
]
Steve Loughran commented on HADOOP-13600:
-
I've just stuck up my nov 10 patch as a PR too. Mine is WiP; I'd stopped as I'd
got to a point where it was too complex. Now we know of the thread pool
problems in the TransferManager we'll have to look at both.
What I have pulled out from mine, which I want to get in first, is having
{{innerRename()}} raise meaningful exceptions, rather than just return
true/false with no details whatsoever. That's part of HADOOP-13823.
Why? So that when I do an S3A-specific committer, it can stop having to deal
with the ambiguity of rename returning false, and instead fail with meaningful
messages. When I start that I'd make {{innerRename()}} package public, or add
some new @Private scoped void renameStrict() call *which would also take some
progressable callback*.
Looking at your code, I like how you rely on AWS callbacks to trigger deletes.
However, it's nice to be able to pool those to avoid throttling from too many
requests; that could be done by (optionally) building up the list and only
triggering a delete when a threshold was reached.
What your patch doesn't have —and I'd planned to but not done myself— was do
async listing. If we retain the current "process 5000, list and repeat"
strategy then the list creates a bottleneck, as may the waiting for all entries
in a single batch to complete (I'm not sure how much that situation arises, it
would if there was, say, a 4GB file and lots of other small files in the tree;
you could block for that 4GB file even while there is more to copy).
That we could maybe ignore. Other issues
# failures. If one copy fails, we'd want to not submit any more, even if
ongoing work is still completed
# queue saturation. Unless there's a separate rename thread pool (my strategy),
we should have some blocking queue of pending copies. Why? Stops all other IO
blocking just because one thread submitted 5000 copy operations.
> S3a rename() to copy files in a directory in parallel
> -
>
> Key: HADOOP-13600
> URL: https://issues.apache.org/jira/browse/HADOOP-13600
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Currently a directory rename does a one-by-one copy, making the request
> O(files * data). If the copy operations were launched in parallel, the
> duration of the copy may be reducable to the duration of the longest copy.
> For a directory with many files, this will be significant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693525#comment-15693525 ] ASF GitHub Bot commented on HADOOP-13600: - GitHub user steveloughran opened a pull request: https://github.com/apache/hadoop/pull/167 HADOOP-13600 starting on parallel rename, still designing code for max parallelism. Even listing and delete calls should be in parallel threads. Really only need to be collecting at the same rate as copies, which is implicitly defined by the rate of keys added to a delete queue You can merge this pull request into a Git repository by running: $ git pull https://github.com/steveloughran/hadoop s3/HADOOOP-13600-rename Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #167 commit 00a0b79481cced4def8734f1aadfb94ef315d737 Author: Steve Loughran Date: 2016-11-10T10:26:34Z HADOOP-13600 starting on parallel rename, still designing code for max parallelism. Even listing and delete calls should be in parallel threads. Indeed: listing could consider doing a pre-emptive call to grab all of the list, though for a bucket with a few million files this would be too expensive. Really only need to be collecting at the same rate as copies, which is implicitly defined by the rate of keys added to a delete queue Change-Id: I906a1a15f3a7567cbff1999236549627859319a5 > S3a rename() to copy files in a directory in parallel > - > > Key: HADOOP-13600 > URL: https://issues.apache.org/jira/browse/HADOOP-13600 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran > > Currently a directory rename does a one-by-one copy, making the request > O(files * data). If the copy operations were launched in parallel, the > duration of the copy may be reducable to the duration of the longest copy. > For a directory with many files, this will be significant -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[
https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693501#comment-15693501
]
Hadoop QA commented on HADOOP-10776:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
34s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
28s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
33s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 4 new + 266 unchanged - 5 fixed = 270 total (was 271) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
45s{color} | {color:red} hadoop-common-project_hadoop-common-jdk1.8.0_111 with
JDK v1.8.0_111 generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
58s{color} | {color:red} hadoop-common-project_hadoop-common-jdk1.7.0_111 with
JDK v1.7.0_111 generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7)
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
20s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-10776 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840421/HADOOP-10776-branch-2-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b8a5ff604d11 3.13.0-36-lowlatency #63-Ub
[jira] [Commented] (HADOOP-13823) s3a rename: fail if dest file exists; throw FNFE if source is missing
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693418#comment-15693418
]
Steve Loughran commented on HADOOP-13823:
-
Also: s3a doesn't raise an exception if the source isn't there. That's at odds
with the FS Specification
> s3a rename: fail if dest file exists; throw FNFE if source is missing
> -
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13823) s3a rename: fail if dest file exists; throw FNFE if source is missing
[
https://issues.apache.org/jira/browse/HADOOP-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13823:
Summary: s3a rename: fail if dest file exists; throw FNFE if source is
missing (was: s3a rename onto existing file must fail)
> s3a rename: fail if dest file exists; throw FNFE if source is missing
> -
>
> Key: HADOOP-13823
> URL: https://issues.apache.org/jira/browse/HADOOP-13823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> HIVE-15199 shows that s3a allows rename onto an existing file, which is
> something HDFS, azure and s3n do not permit (though file:// does). This is
> breaking bits of Hive, is an inconsistency with HDFS and a regression
> compared to s3n semantics.
> I propose: rejecting the rename on a file -> file rename if the destination
> exists (easy) and changing the s3a.xml contract file to declare the behavior
> change; this is needed for
> {{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
> changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A
[ https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693398#comment-15693398 ] Steve Loughran commented on HADOOP-13421: - Listing v2 will also improve LIST availability for object stores with lots of delete markers. > Switch to v2 of the S3 List Objects API in S3A > -- > > Key: HADOOP-13421 > URL: https://issues.apache.org/jira/browse/HADOOP-13421 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steven K. Wong >Priority: Minor > > Unlike [version > 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the > S3 List Objects API, [version > 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by > default does not fetch object owner information, which S3A doesn't need > anyway. By switching to v2, there will be less data to transfer/process. > Also, it should be more robust when listing a versioned bucket with "a large > number of delete markers" ([according to > AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]). > Methods in S3AFileSystem that use this API include: > * getFileStatus(Path) > * innerDelete(Path, boolean) > * innerListStatus(Path) > * innerRename(Path, Path) > Requires AWS SDK 1.10.75 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13075: Parent Issue: HADOOP-13204 (was: HADOOP-11694) > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13695) S3A to use a thread pool for async path operations
[
https://issues.apache.org/jira/browse/HADOOP-13695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13695:
Parent Issue: HADOOP-13204 (was: HADOOP-11694)
> S3A to use a thread pool for async path operations
> --
>
> Key: HADOOP-13695
> URL: https://issues.apache.org/jira/browse/HADOOP-13695
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> S3A path operations are often slow due to directory scanning, mock directory
> create/delete, etc. Many of these can be done asynchronously
> * because deletion is eventually consistent, deleting parent dirs after an
> operation has returned doesn't alter the behaviour, except in the special
> case of : operation failure.
> * scanning for paths/parents of a file in the create operation only needs to
> complete before the close() operation instantiates the object, no need to
> block create().
> * parallelized COPY calls would permit asynchronous rename.
> We could either use the thread pool used for block writes, or somehow isolate
> low cost path ops (GET, DELETE) from the more expensive calls (COPY, PUT) so
> that a thread doing basic IO doesn't block for the duration of the long op.
> Maybe also use {{Semaphore.tryAcquire()}} and only start async work if there
> actually is an idle thread, doing it synchronously if not. Maybe it depends
> on the operation. path query/cleanup before/after a write is something which
> could be scheduled as just more futures to schedule in the block write.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Status: Open (was: Patch Available) > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Attachment: HADOOP-10776-branch-2-002.patch Patch 002; patch 001 with SecurityUtils > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10776) Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
[ https://issues.apache.org/jira/browse/HADOOP-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10776: Status: Patch Available (was: Open) > Open up already widely-used APIs for delegation-token fetching & renewal to > ecosystem projects > -- > > Key: HADOOP-10776 > URL: https://issues.apache.org/jira/browse/HADOOP-10776 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: HADOOP-10776-20160822.txt, > HADOOP-10776-branch-2-002.patch > > > Storm would like to be able to fetch delegation tokens and forward them on to > running topologies so that they can access HDFS (STORM-346). But to do so we > need to open up access to some of APIs. > Most notably FileSystem.addDelegationTokens(), Token.renew, > Credentials.getAllTokens, and UserGroupInformation but there may be others. > At a minimum adding in storm to the list of allowed API users. But ideally > making them public. Restricting access to such important functionality to > just MR really makes secure HDFS inaccessible to anything except MR, or tools > that reuse MR input formats. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13831) Correct check for error code to detect Azure Storage Throttling and provide retries
[ https://issues.apache.org/jira/browse/HADOOP-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693268#comment-15693268 ] Steve Loughran commented on HADOOP-13831: - can you tag which version this surfaces on, component as fs/azure. thanks > Correct check for error code to detect Azure Storage Throttling and provide > retries > --- > > Key: HADOOP-13831 > URL: https://issues.apache.org/jira/browse/HADOOP-13831 > Project: Hadoop Common > Issue Type: Bug >Reporter: Gaurav Kanade > > Azure Storage throttling affects HBase operations such as archiving old > WALS and others. In such cases the storage driver needs to detect and handle > the exception. We put in this logic to do the retries however the condition > to check for the exception is not always met due to inconsistency in which > the manner the error code is passed back. Instead the retry logic should > check for http status code (503) which is more reliable and consistent check -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13091) DistCp masks potential CRC check failures
[
https://issues.apache.org/jira/browse/HADOOP-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693266#comment-15693266
]
Hadoop QA commented on HADOOP-13091:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
3 new + 375 unchanged - 2 fixed = 378 total (was 377) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
53s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13091 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12803138/HADOOP-13091.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fa5077b3efcc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e15c20e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11129/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11129/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11129/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DistCp masks potential CRC check failures
> -
>
> Key: HADOOP-13091
> URL: https://issues.apache.org/jira/browse/HADOOP-13091
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Elliot West
>Assignee: Yiqun Lin
> Attachments: HADOO
[jira] [Commented] (HADOOP-13091) DistCp masks potential CRC check failures
[
https://issues.apache.org/jira/browse/HADOOP-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693158#comment-15693158
]
Elliot West commented on HADOOP-13091:
--
Will this make it into 2.9.0?
> DistCp masks potential CRC check failures
> -
>
> Key: HADOOP-13091
> URL: https://issues.apache.org/jira/browse/HADOOP-13091
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Elliot West
>Assignee: Yiqun Lin
> Attachments: HADOOP-13091.003.patch, HADOOP-13091.004.patch,
> HDFS-10338.001.patch, HDFS-10338.002.patch
>
>
> There appear to be edge cases whereby CRC checks may be circumvented when
> requests for checksums from the source or target file system fail. In this
> event CRCs could differ between the source and target and yet the DistCp copy
> would succeed, even when the 'skip CRC check' option is not being used.
> The code in question is contained in the method
> [{{org.apache.hadoop.tools.util.DistCpUtils#checksumsAreEqual(...)}}|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L457]
> Specifically this code block suggests that if there is a failure when trying
> to read the source or target checksum then the method will return {{true}}
> (i.e. the checksums are equal), implying that the check succeeded. In actual
> fact we just failed to obtain the checksum and could not perform the check.
> {code}
> try {
> sourceChecksum = sourceChecksum != null ? sourceChecksum :
> sourceFS.getFileChecksum(source);
> targetChecksum = targetFS.getFileChecksum(target);
> } catch (IOException e) {
> LOG.error("Unable to retrieve checksum for " + source + " or "
> + target, e);
> }
> return (sourceChecksum == null || targetChecksum == null ||
> sourceChecksum.equals(targetChecksum));
> {code}
> I believe that at the very least the caught {{IOException}} should be
> re-thrown. If this is not deemed desirable then I believe an option
> ({{--strictCrc}}?) should be added to enforce a strict check where we require
> that both the source and target CRCs are retrieved, are not null, and are
> then compared for equality. If for any reason either of the CRCs retrievals
> fail then an exception is thrown.
> Clearly some {{FileSystems}} do not support CRCs and invocations to
> {{FileSystem.getFileChecksum(...)}} return {{null}} in these instances. I
> would suggest that these should fail a strict CRC check to prevent users
> developing a false sense of security in their copy pipeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692770#comment-15692770
]
Hadoop QA commented on HADOOP-13824:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
53s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13824 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840387/HADOOP-13824.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 510af050043e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e15c20e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11128/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11128/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
> Attachments: HA
[jira] [Commented] (HADOOP-11603) Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized
[
https://issues.apache.org/jira/browse/HADOOP-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692720#comment-15692720
]
Hadoop QA commented on HADOOP-11603:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 40 unchanged - 1 fixed = 40 total (was 41) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
50s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-11603 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800995/HADOOP-11603-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4e5707689492 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e15c20e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11127/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11127/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Metric Snapshot log can be changed #MetricsSystemImpl.java since all the
> services will be initialized
> -
>
> Key: HADOOP-11603
> URL: https://issues.apache.org/jira/browse/HADOOP-11603
> Project: Hadoop Common
> Issue Typ
[jira] [Comment Edited] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692630#comment-15692630
]
John Zhuge edited comment on HADOOP-13824 at 11/24/16 8:43 AM:
---
Patch 002:
* Add test utilility class {{SystemErrCapture}} and {{TeePrintStream}}
* Wei-Chiu's suggestion
[~ste...@apache.org], can not use {{ex.toString()}} because it would break
{{testDFSWithInvalidCommmand}}.
was (Author: jzhuge):
Patch 002:
* Add test utilility class {{SystemErrCapture}} and {{TeePrintStream}}
* Wei-Chiu's suggestion
[~ste...@apache.org], can not use {{ex.toString()}} because it would break
{{testDFSWithInvalidCommmand}}
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
> Attachments: HADOOP-13824.001.patch, HADOOP-13824.002.patch
>
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13824:
Attachment: HADOOP-13824.002.patch
Patch 002:
* Add test utilility class {{SystemErrCapture}} and {{TeePrintStream}}
* Wei-Chiu's suggestion
[~ste...@apache.org], can not use {{ex.toString()}} because it would break
{{testDFSWithInvalidCommmand}}
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
> Attachments: HADOOP-13824.001.patch, HADOOP-13824.002.patch
>
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11603) Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized
[
https://issues.apache.org/jira/browse/HADOOP-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-11603:
---
Target Version/s: 2.8.0
Component/s: metrics
Issue Type: Improvement (was: Bug)
LGTM, +1 pending Jenkins.
> Metric Snapshot log can be changed #MetricsSystemImpl.java since all the
> services will be initialized
> -
>
> Key: HADOOP-11603
> URL: https://issues.apache.org/jira/browse/HADOOP-11603
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11603-002.patch, HADOOP-11603.patch,
> defaultmetricIntialize-classes.JPG
>
>
> Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice
> etc... will initialize DefaultMetricsSystem hence following log will be
> logged for all the classes.. Since there is snapshot feature in HDFS ,it's
> better to change the following...
> {code}
> LOG.info("Scheduled snapshot period at "+ period +" second(s).");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
