[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322044#comment-15322044
]
Akira AJISAKA commented on HADOOP-12892:
I noticed
* We need to set the version to 2.9.0 before running create-release in branch-2
* {{2.9.0/CHANGES*.md}} should be in {{$\{basedir\}/src/site/markdown/release}}
but it is in {{patchprocess/}}. It seems that the {{workingDirectory}} does not
work.
{code:title=hadoop-common-project/hadoop-common/pom.xml}
${basedir}/../../dev-support/bin/releasedocmaker
src/site/markdown/release/
{code}
> fix/rewrite create-release
> --
>
> Key: HADOOP-12892
> URL: https://issues.apache.org/jira/browse/HADOOP-12892
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch,
> HADOOP-12892.01.patch, HADOOP-12892.02.patch, HADOOP-12892.03.patch
>
>
> create-release needs some major surgery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[ https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322062#comment-15322062 ] Xiao Chen commented on HADOOP-13251: In the described case, when yarn sends the renew token request, currently there's no way for KMS to figure out it was yarn who sent it - it creates the ugi from the delegation token. ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L341-L345]) This is in comparison with other renewal feature. For example, hdfs does this by getting the remote user from IPC. ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L5213]). Ping [~rkanter] in case anything I said about yarn/dt is not right. So the solution here seems to be providing a way for KMS server to figure out who the actual renewer is. Patch 1 expresses the idea. Cancel works currently since cancel can be done by either a canceler or owner. But IMHO in the yarn case it really should be yarn to cancel the token too. > DelegationTokenAuthenticationHandler should detect actual renewer when renew > token > -- > > Key: HADOOP-13251 > URL: https://issues.apache.org/jira/browse/HADOOP-13251 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > > Turns out KMS delegation token renewal feature (HADOOP-13155) does not work > well with client side impersonation. > In a MR example, an end user (UGI:user) gets all kinds of DTs (with > renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then > renews these DTs as long as the MR jobs are running. But currently, the token > is used at the kms server side to decide the renewer, in which case is always > the token's owner. This ends up rejecting the renew request due to renewer > mismatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[ https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13251: --- Attachment: HADOOP-13251.01.patch > DelegationTokenAuthenticationHandler should detect actual renewer when renew > token > -- > > Key: HADOOP-13251 > URL: https://issues.apache.org/jira/browse/HADOOP-13251 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13251.01.patch > > > Turns out KMS delegation token renewal feature (HADOOP-13155) does not work > well with client side impersonation. > In a MR example, an end user (UGI:user) gets all kinds of DTs (with > renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then > renews these DTs as long as the MR jobs are running. But currently, the token > is used at the kms server side to decide the renewer, in which case is always > the token's owner. This ends up rejecting the renew request due to renewer > mismatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322134#comment-15322134
]
Akira AJISAKA commented on HADOOP-12892:
{{yetus-wrapper}} changes the working directory to {{patchprocess/}} and
execute the script specified by the first argument, so the {{workingDirectory}}
does not work.
{code:title=yetus-wrapper}
pushd "${HADOOP_PATCHPROCESS}" >/dev/null
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}"
exit 1
fi
{code}
We need to copy {{2.9.0/CHANGES*.md}} from {{patchprocess/}}.
> fix/rewrite create-release
> --
>
> Key: HADOOP-12892
> URL: https://issues.apache.org/jira/browse/HADOOP-12892
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch,
> HADOOP-12892.01.patch, HADOOP-12892.02.patch, HADOOP-12892.03.patch
>
>
> create-release needs some major surgery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
Steve Loughran created HADOOP-13252: --- Summary: add logging of what's going on in s3 auth to help debug problems Key: HADOOP-13252 URL: https://issues.apache.org/jira/browse/HADOOP-13252 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.8.0 Reporter: Steve Loughran Priority: Minor We've now got some fairly complex auth mechanisms going on: -hadoop config, KMS, env vars, "none". IF something isn't working, it's going to be a lot harder to debug. I propose *carefully* adding some debug messages to identify which auth provider is doing the auth, so we can see if the env vars were kicking in, sysprops, etc. What we mustn't do is leak any secrets: this should be identifying whether properties and env vars are set, not what their values are. I don't believe that this will generate a security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1531#comment-1531
]
Tsz Wo Nicholas Sze commented on HADOOP-12910:
--
Sorry, please forgive my bad wording. I agree that chaining has its practical
value. I only want to emphasize that chaining can be archived by other mean
once we have async support.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13212) Provide an option to set the socket buffers in S3AFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13212: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-11694 > Provide an option to set the socket buffers in S3AFileSystem > > > Key: HADOOP-13212 > URL: https://issues.apache.org/jira/browse/HADOOP-13212 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Rajesh Balamohan >Priority: Minor > > It should be possible to provide hints about send/receive buffers to > AmazonS3Client via ClientConfiguration. It would be good to expose these > parameters in S3AFileSystem for perf. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12804: Issue Type: Sub-task (was: Bug) Parent: HADOOP-11694 > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322241#comment-15322241
]
Hadoop QA commented on HADOOP-12804:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-12804 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794416/HADOOP-12804-001.patch
|
| JIRA Issue | HADOOP-12804 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9701/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Read Proxy Password from Credential Providers in S3 FileSystem
> --
>
> Key: HADOOP-12804
> URL: https://issues.apache.org/jira/browse/HADOOP-12804
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Larry McCay
>Assignee: Larry McCay
>Priority: Minor
> Attachments: HADOOP-12804-001.patch
>
>
> HADOOP-12548 added credential provider support for the AWS credentials to
> S3FileSystem. This JIRA is for considering the use of the credential
> providers for the proxy password as well.
> Instead of adding the proxy password to the config file directly and in clear
> text, we could provision it in addition to the AWS credentials into a
> credential provider and keep it out of clear text.
> In terms of usage, it could be added to the same credential store as the AWS
> credentials or potentially to a more universally available path - since it is
> the same for everyone. This would however require multiple providers to be
> configured in the provider.path property and more open file permissions on
> the store itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322245#comment-15322245 ] Steve Loughran commented on HADOOP-12804: - Larry, can you sync this up with the current branch-2 code, do a test run, and tell uswhich s3 infrastructure did you test against. Here I'm curious about how proxy testing went before/after as well as direct testing, though that won't be as easy. Also, looking at the code: should the debug statement really be logging the password? Print it's length, maybe —but not the value > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322273#comment-15322273
]
Tsz Wo Nicholas Sze commented on HADOOP-12910:
--
After some thought, copying CompletionStage and CompletableFuture may not be
straightforward since it requires to copy also the dependent classes such as
the classes in the java.util.function package. So returning ListenableFuture
seems like a better choice.
> There is an additional tactical advantage with ListenableFuture. The current
> implementation AsyncGetFuture already extends AbstractFuture that implements
> ListenableFuture. So it seems pretty low hanging. ...
That's correct. The current code already supports ListenableFuture although we
keep the return type as Future.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12892: --- Attachment: HADOOP-12892.02.branch-2.patch Updated the patch. Ran create-release successfully on my MBA. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322342#comment-15322342 ] Hadoop QA commented on HADOOP-12892: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/console in case of problems. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322344#comment-15322344
]
Akira AJISAKA commented on HADOOP-12892:
02.branch-2 patch:
* Removed {{workingDirectory}}
* Copy CHANGES*.md from patchprocess instead of {{workingDirectory}}
* Create the destination directory before copying documents into the directory.
> fix/rewrite create-release
> --
>
> Key: HADOOP-12892
> URL: https://issues.apache.org/jira/browse/HADOOP-12892
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch,
> HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch,
> HADOOP-12892.03.patch
>
>
> create-release needs some major surgery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322370#comment-15322370
]
Akira AJISAKA commented on HADOOP-12892:
FYI: {{create-release --asfrelease}} option requires {{--docker}} option and
the option requires HADOOP-13245.
> fix/rewrite create-release
> --
>
> Key: HADOOP-12892
> URL: https://issues.apache.org/jira/browse/HADOOP-12892
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch,
> HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch,
> HADOOP-12892.03.patch
>
>
> create-release needs some major surgery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322370#comment-15322370
]
Akira AJISAKA edited comment on HADOOP-12892 at 6/9/16 11:25 AM:
-
FYI: {{create-release \-\-asfrelease}} option requires {{--docker}} option and
the option requires HADOOP-13245.
was (Author: ajisakaa):
FYI: {{create-release --asfrelease}} option requires {{--docker}} option and
the option requires HADOOP-13245.
> fix/rewrite create-release
> --
>
> Key: HADOOP-12892
> URL: https://issues.apache.org/jira/browse/HADOOP-12892
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch,
> HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch,
> HADOOP-12892.03.patch
>
>
> create-release needs some major surgery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322374#comment-15322374 ] Larry McCay commented on HADOOP-12804: -- Yikes, [~ste...@apache.org] - I didn't add that but should have caught it! I guess proxy server creds are a bit less sensitive than others but we shouldn't be doing that. > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13003) Add support for pushing metrics to OpenTSDB with Http
[ https://issues.apache.org/jira/browse/HADOOP-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated HADOOP-13003: - Status: In Progress (was: Patch Available) > Add support for pushing metrics to OpenTSDB with Http > - > > Key: HADOOP-13003 > URL: https://issues.apache.org/jira/browse/HADOOP-13003 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Affects Versions: 2.7.1, 2.2.0 > Environment: Add support for pushing metrics to OpenTSDB with Http > Sink >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: metrics > Attachments: HADOOP-13003-v1.patch, HADOOP-13003.patch > > > Add support for pushing metrics to OpenTSDB with Http Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13003) Add support for pushing metrics to OpenTSDB with Http
[ https://issues.apache.org/jira/browse/HADOOP-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated HADOOP-13003: - Status: Patch Available (was: In Progress) > Add support for pushing metrics to OpenTSDB with Http > - > > Key: HADOOP-13003 > URL: https://issues.apache.org/jira/browse/HADOOP-13003 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Affects Versions: 2.7.1, 2.2.0 > Environment: Add support for pushing metrics to OpenTSDB with Http > Sink >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: metrics > Attachments: HADOOP-13003-v1.patch, HADOOP-13003.patch > > > Add support for pushing metrics to OpenTSDB with Http Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
[ https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322385#comment-15322385 ] Larry McCay commented on HADOOP-13252: -- [~ste...@apache.org] - by auth mechanisms, it seems that you mean mechanisms for looking up credentials for auth. Correct? I'd just like to point out that indicating "none" is leaking a secret - whether this be done explicitly or implicitly. We may want to leave this to indicating whether hadoop config of the credentials or a credential provider is being used. Not the level of detail that you are looking for but it would at least point someone with proper permissions to read the config to the right place. > add logging of what's going on in s3 auth to help debug problems > > > Key: HADOOP-13252 > URL: https://issues.apache.org/jira/browse/HADOOP-13252 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > We've now got some fairly complex auth mechanisms going on: -hadoop config, > KMS, env vars, "none". IF something isn't working, it's going to be a lot > harder to debug. > I propose *carefully* adding some debug messages to identify which auth > provider is doing the auth, so we can see if the env vars were kicking in, > sysprops, etc. > What we mustn't do is leak any secrets: this should be identifying whether > properties and env vars are set, not what their values are. I don't believe > that this will generate a security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322405#comment-15322405
]
Hadoop QA commented on HADOOP-12892:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
34s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
50s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
23s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
46s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
21s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
8s{color} | {color:red} The patch generated 5 new + 496 unchanged - 46 fixed =
501 total (was 542) {color} |
| {color:red}-1{color} | {color:red} shelldocs {color} | {color:red} 0m
8s{color} | {color:red} The patch generated 34 new + 14 unchanged - 0 fixed =
48 total (was 14) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-project-dist in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
22s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{col
[jira] [Updated] (HADOOP-9427) use jUnit assumptions to skip platform-specific tests
[ https://issues.apache.org/jira/browse/HADOOP-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HADOOP-9427: -- Status: Open (was: Patch Available) > use jUnit assumptions to skip platform-specific tests > - > > Key: HADOOP-9427 > URL: https://issues.apache.org/jira/browse/HADOOP-9427 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1, 1-win >Reporter: Arpit Agarwal >Assignee: Gergely Novák > Attachments: HADOOP-9427.001.patch, HADOOP-9427.002.patch > > > Certain tests for platform-specific functionality are either executed only on > Windows or bypass on Windows using checks like "if (Path.WINDOWS)" e.g. > TestNativeIO. > Prefer using jUnit assumptions instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9427) use jUnit assumptions to skip platform-specific tests
[ https://issues.apache.org/jira/browse/HADOOP-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HADOOP-9427: -- Status: Patch Available (was: Open) > use jUnit assumptions to skip platform-specific tests > - > > Key: HADOOP-9427 > URL: https://issues.apache.org/jira/browse/HADOOP-9427 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1, 1-win >Reporter: Arpit Agarwal >Assignee: Gergely Novák > Attachments: HADOOP-9427.001.patch, HADOOP-9427.002.patch > > > Certain tests for platform-specific functionality are either executed only on > Windows or bypass on Windows using checks like "if (Path.WINDOWS)" e.g. > TestNativeIO. > Prefer using jUnit assumptions instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322457#comment-15322457 ] Allen Wittenauer commented on HADOOP-12892: --- bq. yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work. It changes it back just prior to execution. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322467#comment-15322467 ] Allen Wittenauer commented on HADOOP-12892: --- (Although, now that releasedocmaker supports a location in yetus 0.3.0, we should probably change the pom.xml to use it.) > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13241) document s3a better
[ https://issues.apache.org/jira/browse/HADOOP-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13241: Status: Open (was: Patch Available) > document s3a better > --- > > Key: HADOOP-13241 > URL: https://issues.apache.org/jira/browse/HADOOP-13241 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13241-branch-2-001.patch > > > s3a can be documented better, things like classpath, troubleshooting, etc. > sit down and do it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322497#comment-15322497
]
Steve Loughran commented on HADOOP-3733:
Can I observer that one of my secret keys does contain a / in it, and I've not
seen this recently
> "s3:" URLs break when Secret Key contains a slash, even if encoded
> --
>
> Key: HADOOP-3733
> URL: https://issues.apache.org/jira/browse/HADOOP-3733
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 0.17.1, 2.0.2-alpha
>Reporter: Stuart Sierra
>Priority: Minor
> Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733.patch,
> hadoop-3733.patch
>
>
> When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line,
> distcp fails if the SECRET contains a slash, even when the slash is
> URL-encoded as %2F.
> Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
> And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> And your bucket is called "mybucket"
> You can URL-encode the Secret KKey as
> Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> But this doesn't work:
> {noformat}
> $ bin/hadoop distcp file:///source
> s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
> 08/07/09 15:05:22 INFO util.CopyFiles:
> destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket:
> mybucket
> org.jets3t.service.S3ServiceException: S3 HEAD request failed.
> ResponseCode=403, ResponseMessage=Forbidden
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
> ...
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message:
> encoding="UTF-8"?>SignatureDoesNotMatchThe
> request signature we calculated does not match the signature you provided.
> Check your key and signing method.
> at
> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13253) KDiag fails if there's a % sign in krb5.conf
Steve Loughran created HADOOP-13253: --- Summary: KDiag fails if there's a % sign in krb5.conf Key: HADOOP-13253 URL: https://issues.apache.org/jira/browse/HADOOP-13253 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.8.0 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor This isn't something that I've seen in the Hadoop code, but as it's surfaced in a functional test in slider's original code (SLIDER-1140), I suspect it'll be there. See that JIRA for the details, essentially: all strings printed that arent format+varags, just simple "message", mustnt go through String.format. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13241) document s3a better
[ https://issues.apache.org/jira/browse/HADOOP-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322505#comment-15322505 ] Steve Loughran commented on HADOOP-13241: - -pulled para on "/" in secrets -pulled duplicate `amazon-core-java-SDK`entry. -sentence starts with "When a partitioned upload is in progress" -mentioned discp and the committer I'll let you cover the script claspath > document s3a better > --- > > Key: HADOOP-13241 > URL: https://issues.apache.org/jira/browse/HADOOP-13241 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13241-branch-2-001.patch > > > s3a can be documented better, things like classpath, troubleshooting, etc. > sit down and do it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13241) document s3a better
[ https://issues.apache.org/jira/browse/HADOOP-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13241: Attachment: HADOOP-13241-branch-2-002.patch Patch 002, with revisions > document s3a better > --- > > Key: HADOOP-13241 > URL: https://issues.apache.org/jira/browse/HADOOP-13241 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13241-branch-2-001.patch, > HADOOP-13241-branch-2-002.patch > > > s3a can be documented better, things like classpath, troubleshooting, etc. > sit down and do it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13241) document s3a better
[ https://issues.apache.org/jira/browse/HADOOP-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13241: Status: Patch Available (was: Open) > document s3a better > --- > > Key: HADOOP-13241 > URL: https://issues.apache.org/jira/browse/HADOOP-13241 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13241-branch-2-001.patch, > HADOOP-13241-branch-2-002.patch > > > s3a can be documented better, things like classpath, troubleshooting, etc. > sit down and do it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13245) Fix up some misc create-release issues
[ https://issues.apache.org/jira/browse/HADOOP-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13245: -- Attachment: HADOOP-13245.02.patch -02: * default SIGN to false * change releasedocmaker to use outputdir > Fix up some misc create-release issues > -- > > Key: HADOOP-13245 > URL: https://issues.apache.org/jira/browse/HADOOP-13245 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13245.00.patch, HADOOP-13245.01.patch, > HADOOP-13245.02.patch > > > 1. Apache Yetus 0.3.0 requires the dateutil.parser module for Python. This > needs to get added to the Dockerfile > 2. Missing -Pdocs so that documentation build is complete -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13245) Fix up some misc create-release issues
[ https://issues.apache.org/jira/browse/HADOOP-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322565#comment-15322565 ] Hadoop QA commented on HADOOP-13245: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9704/console in case of problems. > Fix up some misc create-release issues > -- > > Key: HADOOP-13245 > URL: https://issues.apache.org/jira/browse/HADOOP-13245 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13245.00.patch, HADOOP-13245.01.patch, > HADOOP-13245.02.patch > > > 1. Apache Yetus 0.3.0 requires the dateutil.parser module for Python. This > needs to get added to the Dockerfile > 2. Missing -Pdocs so that documentation build is complete -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13245) Fix up some misc create-release issues
[
https://issues.apache.org/jira/browse/HADOOP-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322566#comment-15322566
]
Hadoop QA commented on HADOOP-13245:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
6s{color} | {color:red} Docker failed to build yetus/hadoop:2c91fd8. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809187/HADOOP-13245.02.patch
|
| JIRA Issue | HADOOP-13245 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9704/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix up some misc create-release issues
> --
>
> Key: HADOOP-13245
> URL: https://issues.apache.org/jira/browse/HADOOP-13245
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Attachments: HADOOP-13245.00.patch, HADOOP-13245.01.patch,
> HADOOP-13245.02.patch
>
>
> 1. Apache Yetus 0.3.0 requires the dateutil.parser module for Python. This
> needs to get added to the Dockerfile
> 2. Missing -Pdocs so that documentation build is complete
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13253) KDiag fails if there's a % sign in krb5.conf
[ https://issues.apache.org/jira/browse/HADOOP-13253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13253: Attachment: HADOOP-13253-branch-2-001.patch Patch 001: derived from the slider patch with tests -> java, formatting logic hardened > KDiag fails if there's a % sign in krb5.conf > > > Key: HADOOP-13253 > URL: https://issues.apache.org/jira/browse/HADOOP-13253 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13253-branch-2-001.patch > > > This isn't something that I've seen in the Hadoop code, but as it's surfaced > in a functional test in slider's original code (SLIDER-1140), I suspect it'll > be there. > See that JIRA for the details, essentially: all strings printed that arent > format+varags, just simple "message", mustnt go through String.format. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12892: --- Status: Open (was: Patch Available) > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12743) Fix git environment check during test-patch
[
https://issues.apache.org/jira/browse/HADOOP-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-12743:
---
Fix Version/s: (was: 3.0.0-alpha1)
2.8.0
Backport this to branch-2 and branch-2.8 because this is needed to backport
HADOOP-12892.
> Fix git environment check during test-patch
> ---
>
> Key: HADOOP-12743
> URL: https://issues.apache.org/jira/browse/HADOOP-12743
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ray Chiang
>Assignee: Allen Wittenauer
> Fix For: 2.8.0
>
> Attachments: HADOOP-12743.00.patch
>
>
> I'm seeing this error when running test-patch in Apache Hadoop:
> {quote}
>
>
> Confirming git environment
>
>
> ERROR: /Users/rchiang/Dev_01/ah_01/patchprocess is not a git repo.
>
>
> {quote}
> From a follow up email:
> it’s a trivial bug in the yetus-wrapper. Missing a popd after extraction.
> Simple fix is to run the command twice. (since the short circuit kicks in
> after yetus is cached).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322604#comment-15322604 ] Akira AJISAKA commented on HADOOP-12892: bq. It changes it back just prior to execution. Thank you for the reply. This is true after HADOOP-12743, which it is not in branch-2. I'll backport this. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13253) KDiag fails if there's a % sign in krb5.conf
[ https://issues.apache.org/jira/browse/HADOOP-13253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13253: Status: Patch Available (was: Open) > KDiag fails if there's a % sign in krb5.conf > > > Key: HADOOP-13253 > URL: https://issues.apache.org/jira/browse/HADOOP-13253 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13253-branch-2-001.patch > > > This isn't something that I've seen in the Hadoop code, but as it's surfaced > in a functional test in slider's original code (SLIDER-1140), I suspect it'll > be there. > See that JIRA for the details, essentially: all strings printed that arent > format+varags, just simple "message", mustnt go through String.format. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[
https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322618#comment-15322618
]
Steve Loughran commented on HADOOP-12537:
-
checkstyle
{code}
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java:22:import
java.nio.file.AccessDeniedException;:8: Unused import -
java.nio.file.AccessDeniedException.
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java:82:
public void TestSTS() throws IOException {:15: Name 'TestSTS' must match
pattern '^[a-z][a-zA-Z0-9]*$'.
{code}
> s3a: Add flag for session ID to allow Amazon STS temporary credentials
> --
>
> Key: HADOOP-12537
> URL: https://issues.apache.org/jira/browse/HADOOP-12537
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Minor
> Attachments: HADOOP-12537-branch-2.005.patch, HADOOP-12537.001.patch,
> HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch,
> HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff
>
>
> Amazon STS allows you to issue temporary access key id / secret key pairs for
> your a user / role. However, using these credentials also requires specifying
> a session ID. There is currently no such configuration property or the
> required code to pass it through to the API (at least not that I can find) in
> any of the S3 connectors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13237) s3a initialization against public bucket fails if caller lacks any credentials
[
https://issues.apache.org/jira/browse/HADOOP-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322621#comment-15322621
]
Steve Loughran commented on HADOOP-13237:
-
LGTM...doing a test run
> s3a initialization against public bucket fails if caller lacks any credentials
> --
>
> Key: HADOOP-13237
> URL: https://issues.apache.org/jira/browse/HADOOP-13237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13237-branch-2.002.patch, HADOOP-13237.001.patch
>
>
> If an S3 bucket is public, anyone should be able to read from it.
> However, you cannot create an s3a client bonded to a public bucket unless you
> have some credentials; the {{doesBucketExist()}} check rejects the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12537: Attachment: HADOOP-12537-branch-2.007.patch HADOOP-12537: patch 007: 006 + checkstyle fixes > s3a: Add flag for session ID to allow Amazon STS temporary credentials > -- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13241) document s3a better
[
https://issues.apache.org/jira/browse/HADOOP-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322653#comment-15322653
]
Hadoop QA commented on HADOOP-13241:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 57s{color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_101 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:babe025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809184/HADOOP-13241-branch-2-002.patch
|
| JIRA Issue | HADOOP-13241 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 4beb67234cc1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Updated] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12537: Status: Open (was: Patch Available) > s3a: Add flag for session ID to allow Amazon STS temporary credentials > -- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13253) KDiag fails if there's a % sign in krb5.conf
[
https://issues.apache.org/jira/browse/HADOOP-13253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322698#comment-15322698
]
Hadoop QA commented on HADOOP-13253:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
12s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
6s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
17s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
37s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:babe025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809189/HADOOP-13253-branch-2-001.patch
|
| JIRA Issue | HADOOP-13253 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 319f65d1a883 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / a086fd9 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 |
| f
[jira] [Commented] (HADOOP-13237) s3a initialization against public bucket fails if caller lacks any credentials
[
https://issues.apache.org/jira/browse/HADOOP-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322701#comment-15322701
]
Steve Loughran commented on HADOOP-13237:
-
+1, works for me.
> s3a initialization against public bucket fails if caller lacks any credentials
> --
>
> Key: HADOOP-13237
> URL: https://issues.apache.org/jira/browse/HADOOP-13237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13237-branch-2.002.patch, HADOOP-13237.001.patch
>
>
> If an S3 bucket is public, anyone should be able to read from it.
> However, you cannot create an s3a client bonded to a public bucket unless you
> have some credentials; the {{doesBucketExist()}} check rejects the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling
[
https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322700#comment-15322700
]
Steve Loughran commented on HADOOP-12878:
-
"*" is certainly something that is used in resource requests...I'd don't see it
being used in any filesystem block location status calls. I suspect that any
long-lived application will be doing hostname matching on the blocks that come
back for locality —and may not work with a "*" (it's not a hostname after all),
or not treat it that well.
Assuming the hostname was configurable: you'd give it a list, it'd return a
random subset of them, people could experiment with "*" and
"offsite.example.org" to see what worked well.
> Impersonate hosts in s3a for better data locality handling
> --
>
> Key: HADOOP-12878
> URL: https://issues.apache.org/jira/browse/HADOOP-12878
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Currently, {{localhost}} is passed as locality for each block, causing all
> blocks involved in job to initially target the same node (RM), before being
> moved by the scheduler (to a rack-local node). This reduces parallelism for
> jobs (with short-lived mappers).
> We should mimic Azures implementation: a config setting
> {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list
> of hostnames in the cluster to return to {{getFileBlockLocations}}.
> Possible optimization: for larger systems, it might be better to return N
> (5?) random hostnames to prevent passing a huge array (the downstream code
> assumes size = O(3)).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12892: --- Attachment: HADOOP-12892.03.branch-2.patch 03.branch-2 patch: After backporting HADOOP-12743, reverted the followings * Removed workingDirectory * Copy CHANGES*.md from patchprocess instead of workingDirectory > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.branch-2.patch, HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13237) s3a initialization against public bucket fails if caller lacks any credentials
[
https://issues.apache.org/jira/browse/HADOOP-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322802#comment-15322802
]
Steve Loughran commented on HADOOP-13237:
-
(commit in progress, just being strict and testing on every branch before
pushing up results)
> s3a initialization against public bucket fails if caller lacks any credentials
> --
>
> Key: HADOOP-13237
> URL: https://issues.apache.org/jira/browse/HADOOP-13237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13237-branch-2.002.patch, HADOOP-13237.001.patch
>
>
> If an S3 bucket is public, anyone should be able to read from it.
> However, you cannot create an s3a client bonded to a public bucket unless you
> have some credentials; the {{doesBucketExist()}} check rejects the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12892: --- Status: Patch Available (was: Open) > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.branch-2.patch, HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13070) classloading isolation improvements for cleaner and stricter dependencies
[
https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322828#comment-15322828
]
Sean Busbey commented on HADOOP-13070:
--
+1 on removing Configuration's dedicated classloader. That simplification helps
limit our pain to the ones java folks expect to have in TCCL.
{quote}
We need to explore an op on that can let you determine the calling class and
only block a user calling class to load a parent class (rule #4). We might be
able to accomplish this by trying to determine the calling class and its
classloader from the stack trace. This is something that the JDK’s ClassLoader
does (via a non‐public JDK‐internal method), and we may be able to implement
something similar.
{quote}
Ugh. this gives me all kinds of bad feels, though I think I might agree. I know
[~ste...@apache.org] has strong feelings on the maintenance burden of this kind
of custom classloader work, so let's ping him early.
If we go down this path, how concerned are we going to be with maintaining
cross-JVM compatibility (vs falling back to some kind of "no isolation"
approach)?
If we're at this point, is just shading every 3rd party dependency we use
easier (barring the usual non-relocatable bits)? That would also prevent
downstream folks from relying on them without a very clear at-your-own-risk
step.
> classloading isolation improvements for cleaner and stricter dependencies
> -
>
> Key: HADOOP-13070
> URL: https://issues.apache.org/jira/browse/HADOOP-13070
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: classloading-improvements-ideas-v.3.pdf,
> classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf
>
>
> Related to HADOOP-11656, we would like to make a number of improvements in
> terms of classloading isolation so that user-code can run safely without
> worrying about dependency collisions with the Hadoop dependencies.
> By the same token, it should raised the quality of the user code and its
> specified classpath so that users get clear signals if they specify incorrect
> classpaths.
> This will contain a proposal that will include several improvements some of
> which may not be backward compatible. As such, it should be targeted to the
> next major revision of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13135) Encounter response code 500 when accessing /metrics endpoint
[
https://issues.apache.org/jira/browse/HADOOP-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322830#comment-15322830
]
Jeremy Carroll commented on HADOOP-13135:
-
I encountered this issue this morning. Only happen with HBase (1.2.1 - Stable).
> Encounter response code 500 when accessing /metrics endpoint
>
>
> Key: HADOOP-13135
> URL: https://issues.apache.org/jira/browse/HADOOP-13135
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ted Yu
>
> When accessing /metrics endpoint on hbase master through hadoop 2.7.1, I got:
> {code}
> HTTP ERROR 500
> Problem accessing /metrics. Reason:
> INTERNAL_SERVER_ERROR
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1029)
> at
> org.apache.hadoop.metrics.MetricsServlet.doGet(MetricsServlet.java:109)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> {code}
> [~ajisakaa] suggested that code 500 should be 404 (NOT FOUND).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322831#comment-15322831 ] Hadoop QA commented on HADOOP-12892: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/console in case of problems. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.branch-2.patch, HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13237) s3a initialization against public bucket fails if caller lacks any credentials
[
https://issues.apache.org/jira/browse/HADOOP-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13237:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> s3a initialization against public bucket fails if caller lacks any credentials
> --
>
> Key: HADOOP-13237
> URL: https://issues.apache.org/jira/browse/HADOOP-13237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13237-branch-2.002.patch, HADOOP-13237.001.patch
>
>
> If an S3 bucket is public, anyone should be able to read from it.
> However, you cannot create an s3a client bonded to a public bucket unless you
> have some credentials; the {{doesBucketExist()}} check rejects the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[ https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13251: --- Status: Patch Available (was: Open) > DelegationTokenAuthenticationHandler should detect actual renewer when renew > token > -- > > Key: HADOOP-13251 > URL: https://issues.apache.org/jira/browse/HADOOP-13251 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13251.01.patch > > > Turns out KMS delegation token renewal feature (HADOOP-13155) does not work > well with client side impersonation. > In a MR example, an end user (UGI:user) gets all kinds of DTs (with > renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then > renews these DTs as long as the MR jobs are running. But currently, the token > is used at the kms server side to decide the renewer, in which case is always > the token's owner. This ends up rejecting the renew request due to renewer > mismatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[
https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322874#comment-15322874
]
Hadoop QA commented on HADOOP-13251:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13251 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809126/HADOOP-13251.01.patch
|
| JIRA Issue | HADOOP-13251 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9707/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DelegationTokenAuthenticationHandler should detect actual renewer when renew
> token
> --
>
> Key: HADOOP-13251
> URL: https://issues.apache.org/jira/browse/HADOOP-13251
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13251.01.patch
>
>
> Turns out KMS delegation token renewal feature (HADOOP-13155) does not work
> well with client side impersonation.
> In a MR example, an end user (UGI:user) gets all kinds of DTs (with
> renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then
> renews these DTs as long as the MR jobs are running. But currently, the token
> is used at the kms server side to decide the renewer, in which case is always
> the token's owner. This ends up rejecting the renew request due to renewer
> mismatch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13237) s3a initialization against public bucket fails if caller lacks any credentials
[
https://issues.apache.org/jira/browse/HADOOP-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322887#comment-15322887
]
Hudson commented on HADOOP-13237:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #9936 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9936/])
HADOOP-13237: s3a initialization against public bucket fails if caller (stevel:
rev 656c460c0e79ee144d6ef48d85cec04a1af3b2cc)
*
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
*
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BasicAWSCredentialsProvider.java
* hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
* hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
*
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
*
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AnonymousAWSCredentialsProvider.java
> s3a initialization against public bucket fails if caller lacks any credentials
> --
>
> Key: HADOOP-13237
> URL: https://issues.apache.org/jira/browse/HADOOP-13237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13237-branch-2.002.patch, HADOOP-13237.001.patch
>
>
> If an S3 bucket is public, anyone should be able to read from it.
> However, you cannot create an s3a client bonded to a public bucket unless you
> have some credentials; the {{doesBucketExist()}} check rejects the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322929#comment-15322929
]
Hadoop QA commented on HADOOP-12892:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
51s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
48s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
10s{color} | {color:red} The patch generated 5 new + 496 unchanged - 46 fixed =
501 total (was 542) {color} |
| {color:red}-1{color} | {color:red} shelldocs {color} | {color:red} 0m
9s{color} | {color:red} The patch generated 34 new + 14 unchanged - 0 fixed =
48 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-project-dist in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color}
[jira] [Updated] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12537: Status: Patch Available (was: Open) > s3a: Add flag for session ID to allow Amazon STS temporary credentials > -- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12943) Add -w -r options in dfs -test command
[
https://issues.apache.org/jira/browse/HADOOP-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322970#comment-15322970
]
Akira AJISAKA commented on HADOOP-12943:
Mostly looks good to me. Thanks [~cheersyang].
{code:Test.java}
} catch (AccessControlException e) {
return false;
} catch (FileNotFoundException e) {
return false;
}
{code}
Would you use multi-catch statement here? I'm +1 if that is addressed.
> Add -w -r options in dfs -test command
> --
>
> Key: HADOOP-12943
> URL: https://issues.apache.org/jira/browse/HADOOP-12943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, scripts, tools
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Fix For: 2.8.0
>
> Attachments: HADOOP-12943.001.patch, HADOOP-12943.002.patch,
> HADOOP-12943.003.patch, HADOOP-12943.004.patch
>
>
> Currently the dfs -test command only supports
> -d, -e, -f, -s, -z
> options. It would be helpful if we add
> -w, -r
> to verify permission of r/w before actual read or write. This will help
> script programming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[ https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13251: --- Attachment: HADOOP-13251.01.patch > DelegationTokenAuthenticationHandler should detect actual renewer when renew > token > -- > > Key: HADOOP-13251 > URL: https://issues.apache.org/jira/browse/HADOOP-13251 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13251.01.patch, HADOOP-13251.01.patch > > > Turns out KMS delegation token renewal feature (HADOOP-13155) does not work > well with client side impersonation. > In a MR example, an end user (UGI:user) gets all kinds of DTs (with > renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then > renews these DTs as long as the MR jobs are running. But currently, the token > is used at the kms server side to decide the renewer, in which case is always > the token's owner. This ends up rejecting the renew request due to renewer > mismatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12943) Add -w -r options in dfs -test command
[
https://issues.apache.org/jira/browse/HADOOP-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322970#comment-15322970
]
Akira AJISAKA edited comment on HADOOP-12943 at 6/9/16 5:49 PM:
Mostly looks good to me. Thanks [~cheersyang].
{code:title=Test.java}
} catch (AccessControlException e) {
return false;
} catch (FileNotFoundException e) {
return false;
}
{code}
Would you use multi-catch statement here? I'm +1 if that is addressed.
was (Author: ajisakaa):
Mostly looks good to me. Thanks [~cheersyang].
{code:Test.java}
} catch (AccessControlException e) {
return false;
} catch (FileNotFoundException e) {
return false;
}
{code}
Would you use multi-catch statement here? I'm +1 if that is addressed.
> Add -w -r options in dfs -test command
> --
>
> Key: HADOOP-12943
> URL: https://issues.apache.org/jira/browse/HADOOP-12943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, scripts, tools
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Fix For: 2.8.0
>
> Attachments: HADOOP-12943.001.patch, HADOOP-12943.002.patch,
> HADOOP-12943.003.patch, HADOOP-12943.004.patch
>
>
> Currently the dfs -test command only supports
> -d, -e, -f, -s, -z
> options. It would be helpful if we add
> -w, -r
> to verify permission of r/w before actual read or write. This will help
> script programming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322984#comment-15322984 ] Akira AJISAKA commented on HADOOP-13175: +1 > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[
https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322991#comment-15322991
]
Hadoop QA commented on HADOOP-13251:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
6s{color} | {color:red} Docker failed to build yetus/hadoop:2c91fd8. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809241/HADOOP-13251.01.patch
|
| JIRA Issue | HADOOP-13251 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9709/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DelegationTokenAuthenticationHandler should detect actual renewer when renew
> token
> --
>
> Key: HADOOP-13251
> URL: https://issues.apache.org/jira/browse/HADOOP-13251
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13251.01.patch, HADOOP-13251.01.patch
>
>
> Turns out KMS delegation token renewal feature (HADOOP-13155) does not work
> well with client side impersonation.
> In a MR example, an end user (UGI:user) gets all kinds of DTs (with
> renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then
> renews these DTs as long as the MR jobs are running. But currently, the token
> is used at the kms server side to decide the renewer, in which case is always
> the token's owner. This ends up rejecting the renew request due to renewer
> mismatch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[ https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13251: --- Attachment: HADOOP-13251.01.patch > DelegationTokenAuthenticationHandler should detect actual renewer when renew > token > -- > > Key: HADOOP-13251 > URL: https://issues.apache.org/jira/browse/HADOOP-13251 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13251.01.patch, HADOOP-13251.01.patch, > HADOOP-13251.01.patch > > > Turns out KMS delegation token renewal feature (HADOOP-13155) does not work > well with client side impersonation. > In a MR example, an end user (UGI:user) gets all kinds of DTs (with > renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then > renews these DTs as long as the MR jobs are running. But currently, the token > is used at the kms server side to decide the renewer, in which case is always > the token's owner. This ends up rejecting the renew request due to renewer > mismatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13251) DelegationTokenAuthenticationHandler should detect actual renewer when renew token
[
https://issues.apache.org/jira/browse/HADOOP-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323033#comment-15323033
]
Hadoop QA commented on HADOOP-13251:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
6s{color} | {color:red} Docker failed to build yetus/hadoop:2c91fd8. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809252/HADOOP-13251.01.patch
|
| JIRA Issue | HADOOP-13251 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9710/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DelegationTokenAuthenticationHandler should detect actual renewer when renew
> token
> --
>
> Key: HADOOP-13251
> URL: https://issues.apache.org/jira/browse/HADOOP-13251
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13251.01.patch, HADOOP-13251.01.patch,
> HADOOP-13251.01.patch
>
>
> Turns out KMS delegation token renewal feature (HADOOP-13155) does not work
> well with client side impersonation.
> In a MR example, an end user (UGI:user) gets all kinds of DTs (with
> renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then
> renews these DTs as long as the MR jobs are running. But currently, the token
> is used at the kms server side to decide the renewer, in which case is always
> the token's owner. This ends up rejecting the renew request due to renewer
> mismatch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-5353) add progress callback feature to the slow FileUtil operations with ability to cancel the work
[
https://issues.apache.org/jira/browse/HADOOP-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pranav Prakash updated HADOOP-5353:
---
Attachment: HADOOP-5353.001.patch
Attached is a preliminary patch that allows one to pass an `IOProgress` object
to the copy methods and poll the transfer status
> add progress callback feature to the slow FileUtil operations with ability to
> cancel the work
> -
>
> Key: HADOOP-5353
> URL: https://issues.apache.org/jira/browse/HADOOP-5353
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 0.21.0
>Reporter: Steve Loughran
>Assignee: Pranav Prakash
>Priority: Minor
> Attachments: HADOOP-5353.000.patch, HADOOP-5353.001.patch
>
>
> This is something only of relevance of people doing front ends to FS
> operations, and as they could take the code in FSUtil and add something with
> this feature, its a blocker to none of them.
> Current FileUtil.copy can take a long time to move large files around, but
> there is no progress indicator to GUIs, or a way to cancel the operation
> mid-way, j interrupting the thread or closing the filesystem.
> I propose a FileIOProgress interface to the copy ops, one that had a single
> method to notify listeners of bytes read and written, and the number of files
> handled.
> {code}
> interface FileIOProgress {
> boolean progress(int files, long bytesRead, long bytesWritten);
> }
> The return value would be true to continue the operation, or false to stop
> the copy and leave the FS in whatever incomplete state it is in currently.
> it could even be fancier: have beginFileOperation and endFileOperation
> callbacks to pass in the name of the current file being worked on, though I
> don't have a personal need for that.
> GUIs could show progress bars and cancel buttons, other tools could use the
> interface to pass any cancellation notice upstream.
> The FileUtil.copy operations would call this interface (blocking) after every
> block copy, so the frequency of invocation would depend on block size and
> network/disk speeds. Which is also why I don't propose having any percentage
> done indicators; it's too hard to predict percentage of time done for
> distributed file IO with any degree of accuracy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13249) RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep
[
https://issues.apache.org/jira/browse/HADOOP-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323041#comment-15323041
]
Jing Zhao commented on HADOOP-13249:
The patch looks good to me overall. Some minor comments:
# The failed unit test is related to the patch
# No need to convert InterruptedIOException into IOException since
{{InterruptedIOException}} is already an {{IOException}}.
{code}
306 throw (IOException)new InterruptedIOException("Retry
interrupted"
307 ).initCause(e);
{code}
Other comments, [~ste...@apache.org]?
> RetryInvocationHandler need wrap InterruptedException in IOException when
> call Thread.sleep
> ---
>
> Key: HADOOP-13249
> URL: https://issues.apache.org/jira/browse/HADOOP-13249
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-13249.000.patch, HADOOP-13249.001.patch
>
>
> RetryInvocationHandler need wrap InterruptedException in IOException when
> call Thread.sleep. Otherwise InterruptedException can't be handled correctly
> by other components such as HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13070) classloading isolation improvements for cleaner and stricter dependencies
[ https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323042#comment-15323042 ] Steve Loughran commented on HADOOP-13070: - custom classloaders tend to cause problems downstream, both maintenance and use. That doesn't mean they don't solve some problems: it's just they are dogs to work with. Little dogs that seem like cute little puppies, which eventually become big ugly beasts you have to take for 3 h walks every day, eats everything in the kitchen, uses your bed as a toilet and, due to its habit of biting small children, is something you are scared of yourself. > classloading isolation improvements for cleaner and stricter dependencies > - > > Key: HADOOP-13070 > URL: https://issues.apache.org/jira/browse/HADOOP-13070 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: classloading-improvements-ideas-v.3.pdf, > classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf > > > Related to HADOOP-11656, we would like to make a number of improvements in > terms of classloading isolation so that user-code can run safely without > worrying about dependency collisions with the Hadoop dependencies. > By the same token, it should raised the quality of the user code and its > specified classpath so that users get clear signals if they specify incorrect > classpaths. > This will contain a proposal that will include several improvements some of > which may not be backward compatible. As such, it should be targeted to the > next major revision of Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13070) classloading isolation improvements for cleaner and stricter dependencies
[ https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323058#comment-15323058 ] Sean Busbey commented on HADOOP-13070: -- that sounds like a -0 rather than a -1? > classloading isolation improvements for cleaner and stricter dependencies > - > > Key: HADOOP-13070 > URL: https://issues.apache.org/jira/browse/HADOOP-13070 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: classloading-improvements-ideas-v.3.pdf, > classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf > > > Related to HADOOP-11656, we would like to make a number of improvements in > terms of classloading isolation so that user-code can run safely without > worrying about dependency collisions with the Hadoop dependencies. > By the same token, it should raised the quality of the user code and its > specified classpath so that users get clear signals if they specify incorrect > classpaths. > This will contain a proposal that will include several improvements some of > which may not be backward compatible. As such, it should be targeted to the > next major revision of Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-5353) add progress callback feature to the slow FileUtil operations with ability to cancel the work
[
https://issues.apache.org/jira/browse/HADOOP-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323082#comment-15323082
]
Steve Loughran commented on HADOOP-5353:
Quick review
* please don't change indentation needlessly, do follow hadoop code layout
rules, import wildcard policy etc.
* {{FileUtil.copy()}} should throw FileNotFoundException when things aren't
there.
* line 482 Have a look inside FileSystem.exists() and you'll see why you could
cut that clause and just go with the else clause
* 499. Exception.toString() over Exception.getMessage
* line 529. I think there should be better handling than "false" if the
destination dir couldn't be created, given its
the complete failure of the operation.
* like 546 & elsewhere. Use try-with-resources now the codebase is Java 7+
{{CommandWithDestination}}: {{TargetFileSystem}} supports try-with-resources
cleanup too. Same for {{writeStreamToFile}}
regarding the test,
# look at [
https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md
] for how to write good asserts.
# In particular, assertTrue and assertFalse must have messages, and if some
probe of exception value/state fails, just throw that exception again.
# There's lots of code in {{ContractTestUtils}} for writing/reading/probing
filesystems
> add progress callback feature to the slow FileUtil operations with ability to
> cancel the work
> -
>
> Key: HADOOP-5353
> URL: https://issues.apache.org/jira/browse/HADOOP-5353
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 0.21.0
>Reporter: Steve Loughran
>Assignee: Pranav Prakash
>Priority: Minor
> Attachments: HADOOP-5353.000.patch, HADOOP-5353.001.patch
>
>
> This is something only of relevance of people doing front ends to FS
> operations, and as they could take the code in FSUtil and add something with
> this feature, its a blocker to none of them.
> Current FileUtil.copy can take a long time to move large files around, but
> there is no progress indicator to GUIs, or a way to cancel the operation
> mid-way, j interrupting the thread or closing the filesystem.
> I propose a FileIOProgress interface to the copy ops, one that had a single
> method to notify listeners of bytes read and written, and the number of files
> handled.
> {code}
> interface FileIOProgress {
> boolean progress(int files, long bytesRead, long bytesWritten);
> }
> The return value would be true to continue the operation, or false to stop
> the copy and leave the FS in whatever incomplete state it is in currently.
> it could even be fancier: have beginFileOperation and endFileOperation
> callbacks to pass in the name of the current file being worked on, though I
> don't have a personal need for that.
> GUIs could show progress bars and cancel buttons, other tools could use the
> interface to pass any cancellation notice upstream.
> The FileUtil.copy operations would call this interface (blocking) after every
> block copy, so the frequency of invocation would depend on block size and
> network/disk speeds. Which is also why I don't propose having any percentage
> done indicators; it's too hard to predict percentage of time done for
> distributed file IO with any degree of accuracy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323118#comment-15323118
]
stack commented on HADOOP-12910:
bq. No, they are the same API. Branch-2 is a simplified version of trunk.
So they are not the same then, one is a 'simplified version'.
I think I can see where you are coming from with your thinking that
chaining/callbacks are nonessential and with your fixation on 90s-era AWT-style
APIs registering listeners (that launches rockets). I was just hoping we could
learn from the past rather than repeat it. I was also hoping that downstreamers
didn't have to perform contortions or rely on a 'library' to fill in missing,
essential pieces.
On the latter suggestions, IMO, the pro for ListenableFuture -- that we'd have
same API in branch-2 and in branch-3 -- outweighs all other concerns. Same for
Deferred which seems nicer but w/ same cons. CompletableFuture is a large
undertaking and going by your diggings, seems hard to do w/ our jdk8. Thanks.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[
https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323126#comment-15323126
]
Hadoop QA commented on HADOOP-12537:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
15s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
34s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
16s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
32s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
23s{color} | {color:red} root: The patch generated 2 new + 7 unchanged - 0
fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
9s{color} | {color:green} hadoop-project in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
39s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{co
[jira] [Commented] (HADOOP-13070) classloading isolation improvements for cleaner and stricter dependencies
[ https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323198#comment-15323198 ] Steve Loughran commented on HADOOP-13070: - correct. I'm not vetoing it. I'm just warning that it's really hard to get right > classloading isolation improvements for cleaner and stricter dependencies > - > > Key: HADOOP-13070 > URL: https://issues.apache.org/jira/browse/HADOOP-13070 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: classloading-improvements-ideas-v.3.pdf, > classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf > > > Related to HADOOP-11656, we would like to make a number of improvements in > terms of classloading isolation so that user-code can run safely without > worrying about dependency collisions with the Hadoop dependencies. > By the same token, it should raised the quality of the user code and its > specified classpath so that users get clear signals if they specify incorrect > classpaths. > This will contain a proposal that will include several improvements some of > which may not be backward compatible. As such, it should be targeted to the > next major revision of Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling
[
https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322700#comment-15322700
]
Steve Loughran edited comment on HADOOP-12878 at 6/9/16 7:47 PM:
-
the star symbol is certainly something that is used in resource requests...I'd
don't see it being used in any filesystem block location status calls. I
suspect that any long-lived application will be doing hostname matching on the
blocks that come back for locality —and may not work with a "*" (it's not a
hostname after all), or not treat it that well.
Assuming the hostname was configurable: you'd give it a list, it'd return a
random subset of them, people could experiment with "*" and
"offsite.example.org" to see what worked well.
was (Author: ste...@apache.org):
"*" is certainly something that is used in resource requests...I'd don't see it
being used in any filesystem block location status calls. I suspect that any
long-lived application will be doing hostname matching on the blocks that come
back for locality —and may not work with a "*" (it's not a hostname after all),
or not treat it that well.
Assuming the hostname was configurable: you'd give it a list, it'd return a
random subset of them, people could experiment with "*" and
"offsite.example.org" to see what worked well.
> Impersonate hosts in s3a for better data locality handling
> --
>
> Key: HADOOP-12878
> URL: https://issues.apache.org/jira/browse/HADOOP-12878
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Currently, {{localhost}} is passed as locality for each block, causing all
> blocks involved in job to initially target the same node (RM), before being
> moved by the scheduler (to a rack-local node). This reduces parallelism for
> jobs (with short-lived mappers).
> We should mimic Azures implementation: a config setting
> {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list
> of hostnames in the cluster to return to {{getFileBlockLocations}}.
> Possible optimization: for larger systems, it might be better to return N
> (5?) random hostnames to prevent passing a huge array (the downstream code
> assumes size = O(3)).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323199#comment-15323199 ] Steve Loughran commented on HADOOP-12537: - +1; I'll fix the second checkstyle in the commit > s3a: Add flag for session ID to allow Amazon STS temporary credentials > -- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13184) Add "Apache" to Hadoop project logo
[ https://issues.apache.org/jira/browse/HADOOP-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323208#comment-15323208 ] Steve Loughran commented on HADOOP-13184: - Option 5 > Add "Apache" to Hadoop project logo > --- > > Key: HADOOP-13184 > URL: https://issues.apache.org/jira/browse/HADOOP-13184 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Abhishek > > Many ASF projects include "Apache" in their logo. We should add it to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323214#comment-15323214
]
Ming Ma commented on HADOOP-12910:
--
The community has added support for classloader isolation so that apps can use
a different version of Guava than the version used by hadoop. If we use Guava
at Hadoop API layer, will that complicate things?
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12537) s3a: Add flag for session ID to allow Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12537: Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks for doing this Sean —a lot of people will appreciate it. Changing the title slightly for the generated release notes. > s3a: Add flag for session ID to allow Amazon STS temporary credentials > -- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12537) S3A to support Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12537: Summary: S3A to support Amazon STS temporary credentials (was: s3a: Add flag for session ID to allow Amazon STS temporary credentials) > S3A to support Amazon STS temporary credentials > --- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323220#comment-15323220 ] Steve Loughran commented on HADOOP-9680: Dmitry, I'm reluctant to go near a jets3t upgrade as there are normally consequences somewhere else. IAM and STS Temporary credential support will be in S3A in Hadoop 2.8.0 > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.1.0-beta, 3.0.0-alpha1 >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323231#comment-15323231
]
Steve Loughran commented on HADOOP-12910:
-
We cannot use anything from Guava in a public API. It will make it impossible
to move off guava, ever.
For something experimental to branch-2, maybe. But for a a public API: never.
I'm with Jitendra's option #1. Something disposable for branch-2 as an
experiment: does it work? Meanwhile trunk can use whatever else works well with
lambda-code.
For that to come off, the trunk API needs active engagement from those
downstream projects to help evolve something which suits them. I'm glad,
therefore, that so many are helping out here
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323237#comment-15323237
]
Siddharth Seth commented on HADOOP-12910:
-
+1 for not using Guava in the public API (unless we are copying the code into
the Hadoop codebase). That ends up potentially limiting guava version changes
in the Hadoop major version.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12537) S3A to support Amazon STS temporary credentials
[ https://issues.apache.org/jira/browse/HADOOP-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323254#comment-15323254 ] Hudson commented on HADOOP-12537: - SUCCESS: Integrated in Hadoop-trunk-Commit #9938 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9938/]) HADOOP-12537 S3A to support Amazon STS temporary credentials. (stevel: rev 31ffaf76f2b6e1fd2a141daa4daaebdfecefe727) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BasicAWSCredentialsProvider.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java * hadoop-tools/hadoop-aws/pom.xml * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/CredentialInitializationException.java * hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * hadoop-project/pom.xml > S3A to support Amazon STS temporary credentials > --- > > Key: HADOOP-12537 > URL: https://issues.apache.org/jira/browse/HADOOP-12537 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-12537-branch-2.005.patch, > HADOOP-12537-branch-2.007.patch, HADOOP-12537.001.patch, > HADOOP-12537.002.patch, HADOOP-12537.003.patch, HADOOP-12537.004.patch, > HADOOP-12537.006.patch, HADOOP-12537.diff, HADOOP-12537.diff > > > Amazon STS allows you to issue temporary access key id / secret key pairs for > your a user / role. However, using these credentials also requires specifying > a session ID. There is currently no such configuration property or the > required code to pass it through to the API (at least not that I can find) in > any of the S3 connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
[
https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323261#comment-15323261
]
Chris Nauroth commented on HADOOP-13252:
One thing that's helpful, and works now without code changes, is to add this to
log4j.properties:
{code}
log4j.logger.com.amazonaws.auth=DEBUG
{code}
Then the AWS SDK will print which AWS credential provider was selected from the
chain, without leaking any secrets.
{code}
> hadoop fs -ls s3a://cnauroth-test-aws-s3a/
16/06/09 13:14:13 DEBUG auth.AWSCredentialsProviderChain: Loading credentials
from BasicAWSCredentialsProvider
{code}
However, if this is more about Hadoop's credential provider API, then I don't
think it has the logging you're looking for yet.
> add logging of what's going on in s3 auth to help debug problems
>
>
> Key: HADOOP-13252
> URL: https://issues.apache.org/jira/browse/HADOOP-13252
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> We've now got some fairly complex auth mechanisms going on: -hadoop config,
> KMS, env vars, "none". IF something isn't working, it's going to be a lot
> harder to debug.
> I propose *carefully* adding some debug messages to identify which auth
> provider is doing the auth, so we can see if the env vars were kicking in,
> sysprops, etc.
> What we mustn't do is leak any secrets: this should be identifying whether
> properties and env vars are set, not what their values are. I don't believe
> that this will generate a security risk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13184) Add "Apache" to Hadoop project logo
[ https://issues.apache.org/jira/browse/HADOOP-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323314#comment-15323314 ] Shane Curcuru commented on HADOOP-13184: Note that technically HADOOP as a name (i.e. without any reference to fonts, colors, or the like) is registered; the logo as a graphical element is not registered. So legally the logo should have a ™, not an ®. If the PMC updates the logo, they could certainly request registering the logo (since this is such a major project), which would let us put the ® there as well. > Add "Apache" to Hadoop project logo > --- > > Key: HADOOP-13184 > URL: https://issues.apache.org/jira/browse/HADOOP-13184 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Abhishek > > Many ASF projects include "Apache" in their logo. We should add it to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
[ https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323316#comment-15323316 ] Steve Loughran commented on HADOOP-13252: - I looked at that, but I also noticed it's logging of auth failures was only exception.getMessage(), not the full stack. > add logging of what's going on in s3 auth to help debug problems > > > Key: HADOOP-13252 > URL: https://issues.apache.org/jira/browse/HADOOP-13252 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > We've now got some fairly complex auth mechanisms going on: -hadoop config, > KMS, env vars, "none". IF something isn't working, it's going to be a lot > harder to debug. > I propose *carefully* adding some debug messages to identify which auth > provider is doing the auth, so we can see if the env vars were kicking in, > sysprops, etc. > What we mustn't do is leak any secrets: this should be identifying whether > properties and env vars are set, not what their values are. I don't believe > that this will generate a security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13175. -- Resolution: Fixed Hadoop Flags: Incompatible change Fix Version/s: 3.0.0-alpha1 Release Note: The hadoop-ant module in hadoop-tools has been removed. Committed to trunk, thanks Chris for the patch, Jason and Akira for reviewing! > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Chris Douglas > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13184) Add "Apache" to Hadoop project logo
[ https://issues.apache.org/jira/browse/HADOOP-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323327#comment-15323327 ] Shane Curcuru commented on HADOOP-13184: I love the idea of adding the "APACHE" to the logo; obviously decisions here are up to the PMC (so my opinion is non-binding). It's a little odd with the feather, since it's a tilted & mirror image feather, but I understand that's the only way it works with your horizontal logo format. But again; up to the PMC. > Add "Apache" to Hadoop project logo > --- > > Key: HADOOP-13184 > URL: https://issues.apache.org/jira/browse/HADOOP-13184 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Abhishek > > Many ASF projects include "Apache" in their logo. We should add it to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323336#comment-15323336 ] Hudson commented on HADOOP-13175: - SUCCESS: Integrated in Hadoop-trunk-Commit #9939 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9939/]) HADOOP-13175. Remove hadoop-ant from hadoop-tools. Contributed by Chris (wang: rev 58c3486850c0503aecdeae8b67bb7e6bc42b4da8) * hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/condition/DfsExists.java * hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/condition/DfsIsDir.java * hadoop-tools/hadoop-ant/pom.xml * hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/DfsTask.java * hadoop-tools/hadoop-ant/src/main/resources/org/apache/hadoop/ant/antlib.xml * hadoop-project/pom.xml * hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/condition/DfsZeroLen.java * hadoop-tools/pom.xml * hadoop-tools/hadoop-tools-dist/pom.xml * hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/condition/DfsBaseConditional.java > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Chris Douglas > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-12893: - Fix Version/s: 3.0.0-alpha1 > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12893.002.patch, HADOOP-12893.003.patch, > HADOOP-12893.004.patch, HADOOP-12893.005.patch, HADOOP-12893.006.patch, > HADOOP-12893.007.patch, HADOOP-12893.008.patch, HADOOP-12893.009.patch, > HADOOP-12893.01.patch, HADOOP-12893.011.patch, HADOOP-12893.10.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323339#comment-15323339 ] Andrew Wang commented on HADOOP-12893: -- Thanks for the great work here everyone, particularly Xiao and Akira. I've committed the 011 patch (with my comment fixup) to trunk. Hoping some others can help with the branch-2 backports. I described what's involved in an earlier comment, but it's basically diffing the list of dependencies with trunk, then updating versions and L&N as necessary. > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12893.002.patch, HADOOP-12893.003.patch, > HADOOP-12893.004.patch, HADOOP-12893.005.patch, HADOOP-12893.006.patch, > HADOOP-12893.007.patch, HADOOP-12893.008.patch, HADOOP-12893.009.patch, > HADOOP-12893.01.patch, HADOOP-12893.011.patch, HADOOP-12893.10.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
[
https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323376#comment-15323376
]
Chris Nauroth commented on HADOOP-13252:
There really isn't very much code to {{AWSCredentialsProviderChain}}:
https://github.com/aws/aws-sdk-java/blob/1.10.6/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java
It's pretty much just a list of the underlying providers and a loop to call
{{getCredentials()}} on each one in sequence. If we'd prefer different
logging, I don't think it would be too cumbersome to make our own chain
implementation and use that instead.
> add logging of what's going on in s3 auth to help debug problems
>
>
> Key: HADOOP-13252
> URL: https://issues.apache.org/jira/browse/HADOOP-13252
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> We've now got some fairly complex auth mechanisms going on: -hadoop config,
> KMS, env vars, "none". IF something isn't working, it's going to be a lot
> harder to debug.
> I propose *carefully* adding some debug messages to identify which auth
> provider is doing the auth, so we can see if the env vars were kicking in,
> sysprops, etc.
> What we mustn't do is leak any secrets: this should be identifying whether
> properties and env vars are set, not what their values are. I don't believe
> that this will generate a security risk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
[ https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323380#comment-15323380 ] Chris Nauroth commented on HADOOP-13252: ...and perhaps simultaneously file a pull request to the AWS SDK so that we can drop our custom version in a later release. > add logging of what's going on in s3 auth to help debug problems > > > Key: HADOOP-13252 > URL: https://issues.apache.org/jira/browse/HADOOP-13252 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > We've now got some fairly complex auth mechanisms going on: -hadoop config, > KMS, env vars, "none". IF something isn't working, it's going to be a lot > harder to debug. > I propose *carefully* adding some debug messages to identify which auth > provider is doing the auth, so we can see if the env vars were kicking in, > sysprops, etc. > What we mustn't do is leak any secrets: this should be identifying whether > properties and env vars are set, not what their values are. I don't believe > that this will generate a security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323379#comment-15323379 ] Hudson commented on HADOOP-12893: - SUCCESS: Integrated in Hadoop-trunk-Commit #9940 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9940/]) HADOOP-12893. Verify LICENSE.txt and NOTICE.txt. Contributed by Xiao (wang: rev e383b732c54c542482b0b836e2d2c46eb49b4e2d) * hadoop-build-tools/pom.xml * LICENSE.txt * hadoop-project/pom.xml * pom.xml * hadoop-project-dist/pom.xml * NOTICE.txt > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12893.002.patch, HADOOP-12893.003.patch, > HADOOP-12893.004.patch, HADOOP-12893.005.patch, HADOOP-12893.006.patch, > HADOOP-12893.007.patch, HADOOP-12893.008.patch, HADOOP-12893.009.patch, > HADOOP-12893.01.patch, HADOOP-12893.011.patch, HADOOP-12893.10.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13254) Make Diskchecker Pluggable
Yufei Gu created HADOOP-13254: - Summary: Make Diskchecker Pluggable Key: HADOOP-13254 URL: https://issues.apache.org/jira/browse/HADOOP-13254 Project: Hadoop Common Issue Type: Bug Reporter: Yufei Gu -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13184) Add "Apache" to Hadoop project logo
[ https://issues.apache.org/jira/browse/HADOOP-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323425#comment-15323425 ] Chris Douglas commented on HADOOP-13184: Good to know. The yellow elephant is registered, right? Do we need to update that graphic? We should probably generate both ™ and ®, while this is active. If/when it's registered, we can swap in the correct logo. > Add "Apache" to Hadoop project logo > --- > > Key: HADOOP-13184 > URL: https://issues.apache.org/jira/browse/HADOOP-13184 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Abhishek > > Many ASF projects include "Apache" in their logo. We should add it to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323424#comment-15323424 ] Wangda Tan commented on HADOOP-12892: - Thanks [~ajisakaa]! I've tried to run create-release on my local box (without --docker) on branch-2. It succeeded and can generate artifacts. I've manually take a look at generated sites, and start a simple YARN cluster runs few distributed shell job, all succeeded. The patch has few conflicts while applying to branch-2.8. Could you update patch for branch-2.8? I can help with verification. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.branch-2.patch, > HADOOP-12892.01.patch, HADOOP-12892.02.branch-2.patch, HADOOP-12892.02.patch, > HADOOP-12892.03.branch-2.patch, HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13254) Make Diskchecker Pluggable
[ https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned HADOOP-13254: --- Assignee: Haibo Chen > Make Diskchecker Pluggable > -- > > Key: HADOOP-13254 > URL: https://issues.apache.org/jira/browse/HADOOP-13254 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yufei Gu >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12666) Support Microsoft Azure Data Lake - as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-12666: --- Resolution: Fixed Fix Version/s: 2.9.0 Release Note: Hadoop now supports integration with Azure Data Lake as an alternative Hadoop-compatible file system. Please refer to the Hadoop site documentation of Azure Data Lake for details on usage and configuration. Status: Resolved (was: Patch Available) I have committed this to trunk and branch-2. [~vishwajeet.dusane], thank you for the contribution. Thanks also to the numerous other contributors who gave thorough code review feedback. > Support Microsoft Azure Data Lake - as a file system in Hadoop > -- > > Key: HADOOP-12666 > URL: https://issues.apache.org/jira/browse/HADOOP-12666 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Fix For: 2.9.0 > > Attachments: Create_Read_Hadoop_Adl_Store_Semantics.pdf, > HADOOP-12666-002.patch, HADOOP-12666-003.patch, HADOOP-12666-004.patch, > HADOOP-12666-005.patch, HADOOP-12666-006.patch, HADOOP-12666-007.patch, > HADOOP-12666-008.patch, HADOOP-12666-009.patch, HADOOP-12666-010.patch, > HADOOP-12666-011.patch, HADOOP-12666-012.patch, HADOOP-12666-013.patch, > HADOOP-12666-014.patch, HADOOP-12666-015.patch, HADOOP-12666-016.patch, > HADOOP-12666-1.patch > > Original Estimate: 336h > Time Spent: 336h > Remaining Estimate: 0h > > h2. Description > This JIRA describes a new file system implementation for accessing Microsoft > Azure Data Lake Store (ADL) from within Hadoop. This would enable existing > Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as > input or output. > > ADL is ultra-high capacity, Optimized for massive throughput with rich > management and security features. More details available at > https://azure.microsoft.com/en-us/services/data-lake-store/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
