[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290306#comment-16290306 ] Bharat Viswanadham commented on HADOOP-9747: [~daryn] One more thing I found is we need to remove testCheckTGTAfterLoginFromSubject from branch-2 patch also, this has been addressed in trunk patch. Because in latest patch we have only simple and kerberos configurations. where as previously we used to have simple, user-kerberos(when login from subject) and keytab-kerberos. user-kerberos app config used to have os_specific_login and krb5login configurations, so if no kerberosprincipal is found it used to use os_specific_login user and loginUserFromSubject() used to pass, but now this will not work, as it will say User cannot be null, as the subject will be empty (as there is no user of type KerberosPrincipal). I am providing multiple comments, as more i am understanding the code, getting more questions, hope you don't mind. Please let me know if my analysis looks correct or am i missing something here. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk.01.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290291#comment-16290291
]
Hudson commented on HADOOP-15039:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13372 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13372/])
Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to (zhengkai.zk: rev
28792b6b7f137df1db58496f27de23bbe99cdfd6)
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestBlockingThreadPoolExecutorService.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BlockingThreadPoolExecutorService.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SemaphoredDelegatingExecutor.java
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. (zhengkai.zk:
rev f86c81d923ecce9d1c9fb691bbc78e93b4a65ae7)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestBlockingThreadPoolExecutorService.java
* (add)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SemaphoredDelegatingExecutor.java
* (delete)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SemaphoredDelegatingExecutor.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (delete)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BlockingThreadPoolExecutorService.java
* (add)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/BlockingThreadPoolExecutorService.java
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290282#comment-16290282
]
Kai Zheng commented on HADOOP-15039:
Just done. The commit was corrected.
Sorry for the inconvenience.
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290276#comment-16290276
]
Kai Zheng commented on HADOOP-15039:
Oops, the commit was incorrect. Thanks [~uncleGen] for the catch. I'll revert
the commit and re-commit the patch.
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14914) Change to a safely casting long to int.
[
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290075#comment-16290075
]
Hudson commented on HADOOP-14914:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13370 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13370/])
HADOOP-14914. Change to a safely casting long to int. Contributed by (cliang:
rev 46e18c8da76ea8d91a16e59ba1154c30f37cb9fd)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockChecksumReconstructor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
> Change to a safely casting long to int.
>
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Fix For: 3.1.0
>
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be
> negative, which is an unexpected behavior. We probably at least want to throw
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to
> replace them, which throws an exception if the value overflows an int.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14914) Change to a safely casting long to int.
[
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290063#comment-16290063
]
Chen Liang commented on HADOOP-14914:
-
I've committed to trunk, thanks Ajay for the contribution!
> Change to a safely casting long to int.
>
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Fix For: 3.1.0
>
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be
> negative, which is an unexpected behavior. We probably at least want to throw
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to
> replace them, which throws an exception if the value overflows an int.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14914) Change to a safely casting long to int.
[
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HADOOP-14914:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
> Change to a safely casting long to int.
>
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Fix For: 3.1.0
>
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be
> negative, which is an unexpected behavior. We probably at least want to throw
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to
> replace them, which throws an exception if the value overflows an int.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14914) Change to a safely casting long to int.
[
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290062#comment-16290062
]
Chen Liang commented on HADOOP-14914:
-
Thanks [~yufeigu] for reporting this and [~ajayydv] for working on it. v002
patch LGTM, I ran the failed tests locally, {{TestContainerLaunch}} failed even
without the patch, the others all passed. I will commit it shortly.
> Change to a safely casting long to int.
>
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be
> negative, which is an unexpected behavior. We probably at least want to throw
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to
> replace them, which throws an exception if the value overflows an int.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15085) Output streams closed with IOUtils suppressing write errors
[
https://issues.apache.org/jira/browse/HADOOP-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289861#comment-16289861
]
Jim Brennan commented on HADOOP-15085:
--
I don't believe the Junit failure (TestZKFailoverController) is related to this
patch.
I did not see this failure when I ran Junit tests locally.
Please review.
> Output streams closed with IOUtils suppressing write errors
> ---
>
> Key: HADOOP-15085
> URL: https://issues.apache.org/jira/browse/HADOOP-15085
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Jim Brennan
> Attachments: HADOOP-15085.001.patch, HADOOP-15085.002.patch,
> HADOOP-15085.003.patch, HADOOP-15085.004.patch, HADOOP-15085.005.patch
>
>
> There are a few places in hadoop-common that are closing an output stream
> with IOUtils.cleanupWithLogger like this:
> {code}
> try {
> ...write to outStream...
> } finally {
> IOUtils.cleanupWithLogger(LOG, outStream);
> }
> {code}
> This suppresses any IOException that occurs during the close() method which
> could lead to partial/corrupted output without throwing a corresponding
> exception. The code should either use try-with-resources or explicitly close
> the stream within the try block so the exception thrown during close() is
> properly propagated as exceptions during write operations are.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289648#comment-16289648
]
Sean Mackrory edited comment on HADOOP-13171 at 12/13/17 6:07 PM:
--
[~ste...@apache.org] Apologies if it's already addressed here in a comment I
missed, but I'm wondering what the reason is for not having stream metrics be
registered in the public metrics? I was interested in collecting some of them
through metrics2, and thought it was an oversight, but the comment in this
patch sounds a bit more intentional...
was (Author: mackrorysd):
[~ste...@apache.org] Apologies if it's already addressed here in a comment I
missed, but I'm wondering what the reason is for not having stream metrics be
registered in the public metrics?
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch,
> HADOOP-13171-branch-2.8-017.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289648#comment-16289648
]
Sean Mackrory commented on HADOOP-13171:
[~ste...@apache.org] Apologies if it's already addressed here in a comment I
missed, but I'm wondering what the reason is for not having stream metrics be
registered in the public metrics?
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch,
> HADOOP-13171-branch-2.8-017.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10054) ViewFsFileStatus.toString() is broken
[
https://issues.apache.org/jira/browse/HADOOP-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289604#comment-16289604
]
Xiaoyu Yao commented on HADOOP-10054:
-
Thanks [~hanishakoneru] for the update. Patch v2 LGTM. +1.
Both of the two failure cases from Jenkins cannot repro on my local machine. I
plan to commit it tomorrow in case other watchers on this ticket have
additional comments.
> ViewFsFileStatus.toString() is broken
> -
>
> Key: HADOOP-10054
> URL: https://issues.apache.org/jira/browse/HADOOP-10054
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.0.5-alpha
>Reporter: Paul Han
>Assignee: Hanisha Koneru
>Priority: Minor
> Attachments: HADOOP-10054.001.patch, HADOOP-10054.002.patch
>
>
> ViewFsFileStatus.toString is broken. Following code snippet :
> {code}
> FileStatus stat= somefunc(); // somefunc() returns an instance of
> ViewFsFileStatus
> System.out.println("path:" + stat.getPath());
> System.out.println(stat.toString());
> {code}
> produces the output:
> {code}
> path:viewfs://x.com/user/X/tmp-48
> ViewFsFileStatus{path=null; isDirectory=false; length=0; replication=0;
> blocksize=0; modification_time=0; access_time=0; owner=; group=;
> permission=rw-rw-rw-; isSymlink=false}
> {code}
> Note that "path=null" is not correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15085) Output streams closed with IOUtils suppressing write errors
[
https://issues.apache.org/jira/browse/HADOOP-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289601#comment-16289601
]
genericqa commented on HADOOP-15085:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
530 unchanged - 1 fixed = 530 total (was 531) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 38s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
54s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15085 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901910/HADOOP-15085.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e74e5a821365 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 880cd75 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC
[jira] [Commented] (HADOOP-14788) Credentials readTokenStorageFile to stop wrapping IOEs in IOEs
[
https://issues.apache.org/jira/browse/HADOOP-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289522#comment-16289522
]
Hanisha Koneru commented on HADOOP-14788:
-
Thanks for working on this [~ajayydv].
In the catch clause in _IOUtils#wrapException_, why are we returning a
{{PathIOException}}? Shouldn't it be an {{IOException}}?
{code}
catch (Exception ex) {
// For subclasses which have no (String) constructor throw IOException
// with wrapped message
return new PathIOException(path, exception);
}
{code}
A tiny nit: In the method description of _wrapException_, "if exception"
string is repeated.
> Credentials readTokenStorageFile to stop wrapping IOEs in IOEs
> --
>
> Key: HADOOP-14788
> URL: https://issues.apache.org/jira/browse/HADOOP-14788
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-14788.001.patch, HADOOP-14788.002.patch,
> HADOOP-14788.003.patch
>
>
> When {{Credentials readTokenStorageFile}} gets an IOE. it catches & wraps
> with the filename, so losing the exception class information.
> Is this needed. or can it pass everything up?
> If it is needed, well, it's a common pattern: wrapping the exception with the
> path & operation. Maybe it's time to add an IOE version of
> {{NetworkUtils.wrapException()}} which handles the broader set of IOEs
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15085) Output streams closed with IOUtils suppressing write errors
[
https://issues.apache.org/jira/browse/HADOOP-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated HADOOP-15085:
-
Attachment: HADOOP-15085.005.patch
Uploaded a new patch that addresses the issues raised in the review.
> Output streams closed with IOUtils suppressing write errors
> ---
>
> Key: HADOOP-15085
> URL: https://issues.apache.org/jira/browse/HADOOP-15085
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Jim Brennan
> Attachments: HADOOP-15085.001.patch, HADOOP-15085.002.patch,
> HADOOP-15085.003.patch, HADOOP-15085.004.patch, HADOOP-15085.005.patch
>
>
> There are a few places in hadoop-common that are closing an output stream
> with IOUtils.cleanupWithLogger like this:
> {code}
> try {
> ...write to outStream...
> } finally {
> IOUtils.cleanupWithLogger(LOG, outStream);
> }
> {code}
> This suppresses any IOException that occurs during the close() method which
> could lead to partial/corrupted output without throwing a corresponding
> exception. The code should either use try-with-resources or explicitly close
> the stream within the try block so the exception thrown during close() is
> properly propagated as exceptions during write operations are.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15116) NPE in ResourceManager when ZooKeeper goes down temporary (HA Mode)
Sunil G created HADOOP-15116:
Summary: NPE in ResourceManager when ZooKeeper goes down temporary
(HA Mode)
Key: HADOOP-15116
URL: https://issues.apache.org/jira/browse/HADOOP-15116
Project: Hadoop Common
Issue Type: Bug
Components: ha
Affects Versions: 3.0.0-beta1
Reporter: Sunil G
In an HA enabled cluster (3.0), we found that RM is failing to start with an
NPE from ActiveStandbyElector. Zookeeper was down at this time, hence client
retries were coming for a while
{code}
2017-12-13 18:21:22,460 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-12-13 18:21:22,544 INFO org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService
failed in state INITED; cause: java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:1039)
at
org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:1036)
at
org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:1101)
at
org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:1093)
at
org.apache.hadoop.ha.ActiveStandbyElector.createWithRetries(ActiveStandbyElector.java:1036)
at
org.apache.hadoop.ha.ActiveStandbyElector.ensureParentZNode(ActiveStandbyElector.java:347)
at
org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.serviceInit(ActiveStandbyElectorBasedElectorService.java:110)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:326)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1420)
2017-12-13 18:21:22,545 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Yielding from election
2017-12-13 18:21:22,545 INFO org.apache.hadoop.service.AbstractService: Service
ResourceManager failed in state INITED; cause: java.lang.NullPointerException
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[
https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289171#comment-16289171
]
genericqa commented on HADOOP-14439:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
18s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14439 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880170/HADOOP-14439-02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 59990a8945c4 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 880cd75 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13825/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13825/artifact/out/patch-compile-hadoop-tools_hadoop-aws.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13825/artifact/out/patch-compile-hadoop-tools_hadoop-aws.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13825/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt
|
| findbugs |
https://builds.apache.org/job/P
[jira] [Resolved] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
[ https://issues.apache.org/jira/browse/HADOOP-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13204. - Resolution: Fixed Fix Version/s: 3.0.0 Moved all open issues as dependencies of/subtasks of HADOOP-14831; all this stuff is done in Hadoop 3.0. Marking as done. Thanks to everyone for their help; made stuff even better here than before. "Misson Accomplished!" > Über-jira: S3a phase III: scale and tuning > -- > > Key: HADOOP-13204 > URL: https://issues.apache.org/jira/browse/HADOOP-13204 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 3.0.0 > > > S3A Phase III work; post 2.8. > Areas could include > * customisation > * performance enhancement > * management and metrics > * error/failure handling > And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
[ https://issues.apache.org/jira/browse/HADOOP-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13204: Target Version/s: 2.9.0, 3.0.0 (was: 3.1.0) > Über-jira: S3a phase III: scale and tuning > -- > > Key: HADOOP-13204 > URL: https://issues.apache.org/jira/browse/HADOOP-13204 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > S3A Phase III work; post 2.8. > Areas could include > * customisation > * performance enhancement > * management and metrics > * error/failure handling > And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[
https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14439:
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-14831
> regression: secret stripping from S3x URIs breaks some downstream code
> --
>
> Key: HADOOP-14439
> URL: https://issues.apache.org/jira/browse/HADOOP-14439
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Spark 2.1
>Reporter: Steve Loughran
>Assignee: Vinayakumar B
>Priority: Minor
> Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch
>
>
> Surfaced in SPARK-20799
> Spark is listing the contents of a path with getFileStatus(path), then
> looking up the path value doing a lookup of the contents.
> Apparently the lookup is failing to find files if you have a secret in the
> key, {{s3a://key:secret@bucket/path}}.
> Presumably this is because the stripped values aren't matching.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11621) s3a doesn't consider blobs with trailing / and content-length >0 as directories
[ https://issues.apache.org/jira/browse/HADOOP-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11621: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14831 > s3a doesn't consider blobs with trailing / and content-length >0 as > directories > --- > > Key: HADOOP-11621 > URL: https://issues.apache.org/jira/browse/HADOOP-11621 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Denis Jannot > > When creating a directory using the AWS Management Console, the > content-length is set to 0 and s3a works fine. > When creating a directory using other tools, like S3Browse, the > content-length is set to 1 and s3a doesn't work: > S3AFileSystem: Found file (with /): real file? should not happen: dir1 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11621) s3a doesn't consider blobs with trailing / and content-length >0 as directories
[ https://issues.apache.org/jira/browse/HADOOP-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289112#comment-16289112 ] Steve Loughran commented on HADOOP-11621: - revisiting this. I think we should move to trailing / is a dir, irrespective of content. When we do bulk delete of parent dir markers, we delete all paths up the tree matching that pattern without looking at them, so sometimes they are treated as dirs, sometimes not. Time to be consistent > s3a doesn't consider blobs with trailing / and content-length >0 as > directories > --- > > Key: HADOOP-11621 > URL: https://issues.apache.org/jira/browse/HADOOP-11621 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Denis Jannot > > When creating a directory using the AWS Management Console, the > content-length is set to 0 and s3a works fine. > When creating a directory using other tools, like S3Browse, the > content-length is set to 1 and s3a doesn't work: > S3AFileSystem: Found file (with /): real file? should not happen: dir1 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14325) Stabilise S3A Server Side Encryption
[ https://issues.apache.org/jira/browse/HADOOP-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289086#comment-16289086 ] Steve Loughran commented on HADOOP-14325: - tagging as dependency of HADOOP-14831. FWIW, I think things are stable > Stabilise S3A Server Side Encryption > > > Key: HADOOP-14325 > URL: https://issues.apache.org/jira/browse/HADOOP-14325 > Project: Hadoop Common > Issue Type: Task > Components: documentation, fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran > > Round off the S3 SSE encryption support with everything needed to safely ship > it. > The core code is in, along with tests, so this covers the details > * docs with examples, including JCEKS files > * keeping secrets secret > * any more tests, including scale ones (huge file, rename) > * I'll add a KMS test to my (github) spark suite -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14762) S3A warning of obsolete encryption key which is never used
[
https://issues.apache.org/jira/browse/HADOOP-14762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14762:
Parent Issue: HADOOP-14831 (was: HADOOP-13204)
> S3A warning of obsolete encryption key which is never used
> --
>
> Key: HADOOP-14762
> URL: https://issues.apache.org/jira/browse/HADOOP-14762
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> During CLI init, I get warned of using an obsolete file.
> {code}
> ./hadoop s3guard import s3a://hwdev-steve-ireland-new
> 2017-08-11 18:08:43,935 DEBUG conf.Configuration: Reloading 3 existing
> configurations
> 2017-08-11 18:08:45,146 INFO Configuration.deprecation:
> fs.s3a.server-side-encryption-key is deprecated. Instead, use
> fs.s3a.server-side-encryption.key
> 2017-08-11 18:08:45,702 INFO s3guard.S3GuardTool: Metadata store
> DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new} is
> initialized.
> ./hadoop s3guard import s3a://hwdev-steve-ireland-new
> 2017-08-11 18:08:43,935 DEBUG conf.Configuration: Reloading 3 existing
> configurations
> 2017-08-11 18:08:45,146 INFO Configuration.deprecation:
> fs.s3a.server-side-encryption-key is deprecated. Instead, use
> fs.s3a.server-side-encryption.key
> 2017-08-11 18:08:45,702 INFO s3guard.S3GuardTool: Metadata store
> DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new} is
> initialized.
> ./hadoop s3guard import s3a://hwdev-steve-ireland-new
> 2017-08-11 18:08:43,935 DEBUG conf.Configuration: Reloading 3 existing
> configurations
> 2017-08-11 18:08:45,146 INFO Configuration.deprecation:
> fs.s3a.server-side-encryption-key is deprecated. Instead, use
> fs.s3a.server-side-encryption.key
> 2017-08-11 18:08:45,702 INFO s3guard.S3GuardTool: Metadata store
> DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new} is
> initialized.
> {code}
> I don't have this setting set as far as I can see, not in an XML file nor any
> jceks file. Maybe its falsely being picked up during the scan for jceks files?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-15113) NPE in S3A getFileStatus: null instrumentation on using closed instance
[
https://issues.apache.org/jira/browse/HADOOP-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-15113 started by Steve Loughran.
---
> NPE in S3A getFileStatus: null instrumentation on using closed instance
> ---
>
> Key: HADOOP-15113
> URL: https://issues.apache.org/jira/browse/HADOOP-15113
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> NPE in getFileStatus in a downstream test of mine; s3a ireland
> {{PathMetadata pm = metadataStore.get(path, needEmptyDirectoryFlag);. }}
> Something up with the bucket config?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15107 started by Steve Loughran. --- > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. > I'm now not sure that the magic committer works. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
