[jira] [Commented] (HADOOP-14844) Remove requirement to specify TenantGuid for MSI Token Provider
[ https://issues.apache.org/jira/browse/HADOOP-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156515#comment-16156515 ] John Zhuge commented on HADOOP-14844: - This is a nice enhancement, [~ASikaria]! The puts MSI on par with AWS Instance Profile in ease of use. > Remove requirement to specify TenantGuid for MSI Token Provider > --- > > Key: HADOOP-14844 > URL: https://issues.apache.org/jira/browse/HADOOP-14844 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: HADOOP-14844.001.patch > > > The MSI identity extension on Azure VMs has removed the need to specify the > tenant guid as part of the request to retrieve token from MSI service on the > local VM. This means the tenant guid configuration parameter is not needed > anymore. This change removes the redundant configuration parameter. > It also makes the port number optional - if not specified, then the default > port is used by the ADLS SDK (happens to be 50342, but that is transparent to > Hadoop code). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14844) Remove requirement to specify TenantGuid for MSI Token Provider
[ https://issues.apache.org/jira/browse/HADOOP-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-14844: --- Assignee: Atul Sikaria > Remove requirement to specify TenantGuid for MSI Token Provider > --- > > Key: HADOOP-14844 > URL: https://issues.apache.org/jira/browse/HADOOP-14844 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: HADOOP-14844.001.patch > > > The MSI identity extension on Azure VMs has removed the need to specify the > tenant guid as part of the request to retrieve token from MSI service on the > local VM. This means the tenant guid configuration parameter is not needed > anymore. This change removes the redundant configuration parameter. > It also makes the port number optional - if not specified, then the default > port is used by the ADLS SDK (happens to be 50342, but that is transparent to > Hadoop code). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14808) Hadoop keychain
[
https://issues.apache.org/jira/browse/HADOOP-14808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156417#comment-16156417
]
Hadoop QA commented on HADOOP-14808:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} root: The patch generated 0 new + 354 unchanged - 3
fixed = 354 total (was 357) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 16s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
42s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14808 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885725/HADOOP-14808.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 692796400ee3 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b6e7d13 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13184/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Assigned] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HADOOP-14238: --- Assignee: Bharat Viswanadham > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Assignee: Bharat Viswanadham >Priority: Blocker > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14089) Shaded Hadoop client runtime includes non-shaded classes
[
https://issues.apache.org/jira/browse/HADOOP-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156391#comment-16156391
]
Sean Busbey commented on HADOOP-14089:
--
In the middle of working through rationalizing what I see in the runtime jar.
I'll post a WIP once I have things either relocated or exposed in the pom (even
if we might then later relocate instead of exposing in the pom). Sound
reasonable?
> Shaded Hadoop client runtime includes non-shaded classes
>
>
> Key: HADOOP-14089
> URL: https://issues.apache.org/jira/browse/HADOOP-14089
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: David Phillips
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HADOOP-14089.WIP.0.patch
>
>
> The jar includes things like {{assets}}, {{okio}}, {{javax/annotation}},
> {{javax/ws}}, {{mozilla}}, etc.
> An easy way to verify this is to look at the contents of the jar:
> {code}
> jar tf hadoop-client-runtime-xxx.jar | sort | grep -v '^org/apache/hadoop'
> {code}
> For standard dependencies, such as the JSR 305 {{javax.annotation}} or JAX-RS
> {{javax.ws}}, it makes sense for those to be normal dependencies in the POM
> -- they are standard, so version conflicts shouldn't be a problem. The JSR
> 305 annotations can be {{true}} since they aren't needed
> at runtime (this is what Guava does).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14808) Hadoop keychain
[
https://issues.apache.org/jira/browse/HADOOP-14808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14808:
Attachment: HADOOP-14808.003.patch
Patch 003
* Suppress checkstyle
* Fix findbugs
> Hadoop keychain
> ---
>
> Key: HADOOP-14808
> URL: https://issues.apache.org/jira/browse/HADOOP-14808
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.7.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14808.001.patch, HADOOP-14808.002.patch,
> HADOOP-14808.003.patch
>
>
> Extend the idea from HADOOP-6520 "UGI should load tokens from the
> environment" to a generic lightweight "keychain" design. Load keys (secrets)
> into a keychain in UGI (secret map) at startup. YARN will distribute them
> securely into each container. The Hadoop code running in the container can
> then retrieve the credentials from UGI.
> The use case is Bring Your Own Key (BYOK) credentials for cloud connectors
> (adl, wasb, s3a, etc.), while Hadoop authentication is still Kerberos. No
> configuration change, no admin involved. It will support YARN applications
> initially, e.g., DistCp, Tera Suite, Spark-on-Yarn, etc.
> Implementation is surprisingly simple because almost all pieces are in place:
> * Retrieve secrets from UGI using {{conf.getPassword}} backed by the existing
> Credential Provider class {{UserProvider}}
> * Reuse Credential Provider classes and interface to define local permanent
> or transient credential store, e.g., {{LocalJavaKeyStoreProvider}}
> * New: create a new transient Credential Provider that logs into AAD with
> username/password or device code, and then put the Client ID and Refresh
> Token into the keychain
> * New: create a new permanent Credential Provider based on Hadoop
> configuration XML, for dev/testing purpose.
> Links
> * HADOOP-11766 Generic token authentication support for Hadoop
> * HADOOP-11744 Support OAuth2 in Hadoop
> * HADOOP-10959 A Kerberos based token authentication approach
> * HADOOP-9392 Token based authentication and Single Sign On
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156339#comment-16156339 ] Tsuyoshi Ozawa commented on HADOOP-14238: - [~andrew.wang] [~vinodkv] As far as I know and as [~bharatviswa] mentioned, it would be ideal to substitute Guava's Supplier with java's one. > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Blocker > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned HADOOP-14238: --- Assignee: (was: Tsuyoshi Ozawa) > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Blocker > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14844) Remove requirement to specify TenantGuid for MSI Token Provider
[ https://issues.apache.org/jira/browse/HADOOP-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atul Sikaria updated HADOOP-14844: -- Attachment: HADOOP-14844.001.patch > Remove requirement to specify TenantGuid for MSI Token Provider > --- > > Key: HADOOP-14844 > URL: https://issues.apache.org/jira/browse/HADOOP-14844 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria > Attachments: HADOOP-14844.001.patch > > > The MSI identity extension on Azure VMs has removed the need to specify the > tenant guid as part of the request to retrieve token from MSI service on the > local VM. This means the tenant guid configuration parameter is not needed > anymore. This change removes the redundant configuration parameter. > It also makes the port number optional - if not specified, then the default > port is used by the ADLS SDK (happens to be 50342, but that is transparent to > Hadoop code). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14844) Remove requirement to specify TenantGuid for MSI Token Provider
Atul Sikaria created HADOOP-14844: - Summary: Remove requirement to specify TenantGuid for MSI Token Provider Key: HADOOP-14844 URL: https://issues.apache.org/jira/browse/HADOOP-14844 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Reporter: Atul Sikaria The MSI identity extension on Azure VMs has removed the need to specify the tenant guid as part of the request to retrieve token from MSI service on the local VM. This means the tenant guid configuration parameter is not needed anymore. This change removes the redundant configuration parameter. It also makes the port number optional - if not specified, then the default port is used by the ADLS SDK (happens to be 50342, but that is transparent to Hadoop code). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156312#comment-16156312
]
Kai Zheng commented on HADOOP-12862:
Sorry for the late, pretty busy with EC related issues.
The issue was well documented in the description and I totally agree. The patch
looks good overall, with some comments:
1. {{setSslConf}} could be => {{loadSslConf}}.
2. Writing some plain password in configuration file should be discouraged, if
allowed; could we go for the password from reading the file first?
{code}
+ hadoop.security.group.mapping.ldap.ssl.keystore.password
+
+
+Password for LDAP SSL keystore. If this value is empty, Hadoop will attempt
+to read the password from the file in
+hadoop.security.group.mapping.ldap.ssl.keystore.password.file.
+
+
{code}
3. Lots of _empty_ default values defined here. If we don't really appreciate
or favor the *empty* value at all, I guess we could avoid the overhead of
defining them at all.
{code}
public static final String LDAP_KEYSTORE_PASSWORD_DEFAULT = "";
..
public static final String LDAP_KEYSTORE_PASSWORD_FILE_DEFAULT = "";
..
+ /*
+ * Password for the keystore
+ */
+ public static final String LDAP_TRUSTSTORE_PASSWORD_KEY =
+ LDAP_CONFIG_PREFIX +".ssl.truststore.password";
+ public static final String LDAP_TRUSTSTORE_PASSWORD_DEFAULT = "";
+
+ public static final String LDAP_TRUSTSTORE_PASSWORD_FILE_KEY =
+ LDAP_TRUSTSTORE_PASSWORD_KEY + ".file";
+ public static final String LDAP_TRUSTSTORE_PASSWORD_FILE_DEFAULT = "";
+
{code}
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156290#comment-16156290 ] Bharat Viswanadham edited comment on HADOOP-14771 at 9/7/17 1:32 AM: - Andrew Wang Vinod Kumar Vavilapalli [~busbey] Can we use java8 supplier here, instead of guava to avoid this issue? was (Author: bharatviswa): Andrew Wang Vinod Kumar Vavilapalli [~busbey]Can we use java8 supplier here, instead of guava to avoid this issue? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156290#comment-16156290 ] Bharat Viswanadham edited comment on HADOOP-14771 at 9/7/17 1:32 AM: - [~andrew.wang] [~vinodkv][~busbey] Can we use java8 supplier here, instead of guava to avoid this issue? was (Author: bharatviswa): Andrew Wang Vinod Kumar Vavilapalli [~busbey] Can we use java8 supplier here, instead of guava to avoid this issue? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156290#comment-16156290 ] Bharat Viswanadham commented on HADOOP-14771: - Andrew Wang Vinod Kumar Vavilapalli [~busbey]Can we use java8 supplier here, instead of guava to avoid this issue? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156288#comment-16156288 ] Bharat Viswanadham commented on HADOOP-14238: - [~andrew.wang] [~vinodkv] Can we use java8 supplier here, instead of guava to avoid this issue? > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Assignee: Tsuyoshi Ozawa >Priority: Blocker > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-14738: - Assignee: Aaron Fabbri (was: Steve Loughran) > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs > -- > > Key: HADOOP-14738 > URL: https://issues.apache.org/jira/browse/HADOOP-14738 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-14738-002.patch, HADOOP-14738-003.patch, > HADOOP-14739-001.patch > > > We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf > since Hadoop 2.8 > It's now time to kill S3N off, remove the source, the tests, the transitive > dependencies. > I propose that in Hadoop 3.0 beta we tell people off from using it, and link > to a doc page (wiki?) about how to migrate (Change URLs, update config ops). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14835) mvn site build throws SAX errors
[ https://issues.apache.org/jira/browse/HADOOP-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156235#comment-16156235 ] Allen Wittenauer commented on HADOOP-14835: --- bq. Alternatively, we cut our losses and get rid of JDiff and provide a JACC report instead. Funny you mention that. When I first looked at this issue, I wondered when jdiff had last been updated. Then I started looking at alternatives. japicmp (http://siom79.github.io/japicmp/) looks like it might be a better replacement. The fact it ships with a maven plugin, has filters, etc, is a HUGE win. But I didn't spend a lot of time playing with it. > mvn site build throws SAX errors > > > Key: HADOOP-14835 > URL: https://issues.apache.org/jira/browse/HADOOP-14835 > Project: Hadoop Common > Issue Type: Bug > Components: build, site >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Andrew Wang >Priority: Critical > Attachments: HADOOP-14835.001.patch > > > Running mvn install site site:stage -DskipTests -Pdist,src > -Preleasedocs,docs results in a stack trace when run on a fresh .m2 > directory. It appears to be coming from the jdiff doclets in the annotations > code. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14089) Shaded Hadoop client runtime includes non-shaded classes
[
https://issues.apache.org/jira/browse/HADOOP-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156232#comment-16156232
]
Bharat Viswanadham commented on HADOOP-14089:
-
[~busbey]
Can I do any help testing the changes?
> Shaded Hadoop client runtime includes non-shaded classes
>
>
> Key: HADOOP-14089
> URL: https://issues.apache.org/jira/browse/HADOOP-14089
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: David Phillips
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HADOOP-14089.WIP.0.patch
>
>
> The jar includes things like {{assets}}, {{okio}}, {{javax/annotation}},
> {{javax/ws}}, {{mozilla}}, etc.
> An easy way to verify this is to look at the contents of the jar:
> {code}
> jar tf hadoop-client-runtime-xxx.jar | sort | grep -v '^org/apache/hadoop'
> {code}
> For standard dependencies, such as the JSR 305 {{javax.annotation}} or JAX-RS
> {{javax.ws}}, it makes sense for those to be normal dependencies in the POM
> -- they are standard, so version conflicts shouldn't be a problem. The JSR
> 305 annotations can be {{true}} since they aren't needed
> at runtime (this is what Guava does).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14835) mvn site build throws SAX errors
[ https://issues.apache.org/jira/browse/HADOOP-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156196#comment-16156196 ] Andrew Wang commented on HADOOP-14835: -- The issue is that I added the xerces dependency under the "docs" profile, which means it gets pulled into the hadoop artifacts whenever the docs profile is on. We don't want this. I don't know how to do this in Maven without splitting jdiff generation into a new module. Anyway, we already do the release build in two passes: artifacts, then site. The simplest fix is to not build the shaded artifacts when docs is enabled. Maven again makes it difficult to do this automatically since you can't chain profile activations; the quick fix is to pass "-DskipShade" when we do a docs/site build. This also requires a BUILDING.txt update to make it clear that the site needs to be built in a second pass, as ugly as that is. Alternatively, we cut our losses and get rid of JDiff and provide a JACC report instead. > mvn site build throws SAX errors > > > Key: HADOOP-14835 > URL: https://issues.apache.org/jira/browse/HADOOP-14835 > Project: Hadoop Common > Issue Type: Bug > Components: build, site >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Andrew Wang >Priority: Critical > Attachments: HADOOP-14835.001.patch > > > Running mvn install site site:stage -DskipTests -Pdist,src > -Preleasedocs,docs results in a stack trace when run on a fresh .m2 > directory. It appears to be coming from the jdiff doclets in the annotations > code. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156147#comment-16156147
]
Xiao Chen commented on HADOOP-14841:
Thank you [~ste...@apache.org] for sharing a similar story. Do you have more
information on how that issue is identified and fixed?
Will update the patch to address your comment soon, though it seems we're not
having consensus on where the issue is yet. Hoping the amazon story would help
shed some light here too.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-14841.01.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
>
[jira] [Updated] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A
[ https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13421: -- Attachment: HADOOP-13421.003.patch Attaching v3 patch. Changes from v2: - Use an integer instead of boolean for config as suggested by [~ste...@apache.org]. - Skip testInconsistentS3ClientDeletes() test case when v1 list is configured. - Checkstyle cleanups. Testing (in us-west-2): - All integration tests w/ v2 - All integration tests w/ v2 + s3guard - All integration tests w/ v1 configured. No failures. > Switch to v2 of the S3 List Objects API in S3A > -- > > Key: HADOOP-13421 > URL: https://issues.apache.org/jira/browse/HADOOP-13421 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steven K. Wong >Assignee: Aaron Fabbri >Priority: Minor > Attachments: HADOOP-13421.002.patch, HADOOP-13421.003.patch, > HADOOP-13421-HADOOP-13345.001.patch > > > Unlike [version > 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the > S3 List Objects API, [version > 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by > default does not fetch object owner information, which S3A doesn't need > anyway. By switching to v2, there will be less data to transfer/process. > Also, it should be more robust when listing a versioned bucket with "a large > number of delete markers" ([according to > AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]). > Methods in S3AFileSystem that use this API include: > * getFileStatus(Path) > * innerDelete(Path, boolean) > * innerListStatus(Path) > * innerRename(Path, Path) > Requires AWS SDK 1.10.75 or later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14843) FsPermission symbolic parsing failed to detect invalid argument
[ https://issues.apache.org/jira/browse/HADOOP-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-14843: Status: Patch Available (was: Open) > FsPermission symbolic parsing failed to detect invalid argument > --- > > Key: HADOOP-14843 > URL: https://issues.apache.org/jira/browse/HADOOP-14843 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.1, 2.7.4 >Reporter: Jason Lowe >Assignee: Bharat Viswanadham > Attachments: HADOOP-14843.patch > > > A user misunderstood the syntax format for the FsPermission symbolic > constructor and passed the argument "-rwr" instead of "u=rw,g=r". In 2.7 and > earlier this was silently misinterpreted as mode 0777 and in 2.8 it oddly > became mode . In either case FsPermission should have flagged "-rwr" as > an invalid argument rather than silently misinterpreting it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14843) FsPermission symbolic parsing failed to detect invalid argument
[ https://issues.apache.org/jira/browse/HADOOP-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-14843: Attachment: HADOOP-14843.patch > FsPermission symbolic parsing failed to detect invalid argument > --- > > Key: HADOOP-14843 > URL: https://issues.apache.org/jira/browse/HADOOP-14843 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.4, 2.8.1 >Reporter: Jason Lowe >Assignee: Bharat Viswanadham > Attachments: HADOOP-14843.patch > > > A user misunderstood the syntax format for the FsPermission symbolic > constructor and passed the argument "-rwr" instead of "u=rw,g=r". In 2.7 and > earlier this was silently misinterpreted as mode 0777 and in 2.8 it oddly > became mode . In either case FsPermission should have flagged "-rwr" as > an invalid argument rather than silently misinterpreting it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156058#comment-16156058
]
Hadoop QA commented on HADOOP-13282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 6
new + 5 unchanged - 0 fixed = 11 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
28s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-13282 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885669/HADOOP-13282-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 06203f8ba115 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 704267c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13181/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13181/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13181/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156055#comment-16156055
]
Erik Krogen commented on HADOOP-14827:
--
Thank you very much Jason! Appreciate it.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 2.7.5
>
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-14827:
Fix Version/s: 2.7.5
2.8.3
I committed this to branch-2.8 and branch-2.7 as well.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 2.7.5
>
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156043#comment-16156043
]
Erik Krogen edited comment on HADOOP-14827 at 9/6/17 9:15 PM:
--
[~jlowe] thank you for the review and the commit! Any chance you would be
willing to bring it to 2.7, 2.8? I am relying on this for the unit test in
HDFS-12323 which I think is a severe enough bug that it needs to go to all of
the release lines. It applies cleanly.
was (Author: xkrogen):
[~jlowe] thank you for the review and the commit! Any chance you would be
willing to bring it to 2.7, 2.8? I am relying on this for the unit test in
HDFS-12323 which I think is a severe enough bug that it needs to go to all of
the release lines.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156043#comment-16156043
]
Erik Krogen commented on HADOOP-14827:
--
[~jlowe] thank you for the review and the commit! Any chance you would be
willing to bring it to 2.7, 2.8? I am relying on this for the unit test in
HDFS-12323 which I think is a severe enough bug that it needs to go to all of
the release lines.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-14827:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
Thanks to [~xkrogen] for the contribution and to [~hanishakoneru] for
additional review! I committed this to trunk, branch-3.0, and branch-2.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156032#comment-16156032
]
Jason Lowe commented on HADOOP-14827:
-
Thanks for the patch, Erik! Agree the unit test failures are unrelated.
TestZKFailoverController and TestShellBasedUnixGroupsMapping pass for me
locally with the patch applied.
+1 for the patch. Committing this.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156031#comment-16156031 ] Steve Loughran commented on HADOOP-14220: - bq. On that topic, is there a way to mark CLI interface as unstable? mention it in the docs :) you are right about the troubleshooting docs, I'll do that. But only after I've got the doc reorg of HADOOP-14220 in. I'll be adding some more CLI options related to the committer next, I think I'll want to list & drop multipart uploads under a path, maybe do some more low-levelness. And at some point soon we'll need to think of a "hadoop cloudstore" CLI entry point for the stores & options related to them, such as my HADOOP-14766 util > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156026#comment-16156026
]
Steve Loughran commented on HADOOP-14841:
-
We've seen funnies in the amazon SDK parse about misinterpreting connections as
JSON parse problems. Like your code to replicate this problem.
Can you wrap the {{MSJSONFaultInjector.instance}} field with a setter, have a
Preconditions.checkNotNull() before setting the value. Just because the code is
passed in the production code, and I don't want it to be another way for an NPE
to sneak in, somehow
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-14841.01.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
>
[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156001#comment-16156001
]
Aaron Fabbri edited comment on HADOOP-14220 at 9/6/17 8:38 PM:
---
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding) assuming yetus is happy. Minor suggestions below that you could
roll in before committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
typo at end.
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
was (Author: fabbri):
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
typo at end.
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch,
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch,
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose
> problems for a specific (named) s3a url. This is something which can be
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket
> overrides: s3guard metastore setup, autocreate, capacity,
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156001#comment-16156001
]
Aaron Fabbri edited comment on HADOOP-14220 at 9/6/17 8:37 PM:
---
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
typo at end.
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
was (Author: fabbri):
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch,
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch,
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose
> problems for a specific (named) s3a url. This is something which can be
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket
> overrides: s3guard metastore setup, autocreate, capacity,
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156004#comment-16156004
]
Steve Loughran commented on HADOOP-13282:
-
As an aside, this makes getFileChecksum() a faster way to look for exactly a
file existing at a path than getFileStatus, which does 2 more HTTP requests
before concluding that it is missing, and so raising an exception.
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13282-001.patch
>
>
> If the etags of blobs were exported via {{getFileChecksum()}}, it'd be
> possible to probe for a blob being in sync with a local file. Distcp could
> use this to decide whether to skip a file or not.
> Now, there's a problem there: distcp needs source and dest filesystems to
> implement the same algorithm. It'd only work out the box if you were copying
> between S3 instances. There are also quirks with encryption and multipart:
> [s3
> docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
> At the very least, it's something which could be used when indexing the FS,
> to check for changes later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156001#comment-16156001
]
Aaron Fabbri commented on HADOOP-14220:
---
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch,
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch,
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose
> problems for a specific (named) s3a url. This is something which can be
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket
> overrides: s3guard metastore setup, autocreate, capacity,
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14688) Intern strings in KeyVersion and EncryptedKeyVersion
[
https://issues.apache.org/jira/browse/HADOOP-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156000#comment-16156000
]
Xiao Chen commented on HADOOP-14688:
Thanks Wei-Chiu for review and commit. Also thanks Daryn and Misha for
commenting.
> Intern strings in KeyVersion and EncryptedKeyVersion
>
>
> Key: HADOOP-14688
> URL: https://issues.apache.org/jira/browse/HADOOP-14688
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: GC root of the String.png, HADOOP-14688.01.patch,
> heapdump analysis.png, jxray.report
>
>
> This is inspired by [~mi...@cloudera.com]'s work on HDFS-11383.
> The key names and key version names are usually the same for a bunch of
> {{KeyVersion}} and {{EncryptedKeyVersion}}. We should not create duplicate
> objects for them.
> This is more important to HDFS-10899, where we try to re-encrypt all files'
> EDEKs in a given EZ. Those EDEKs all has the same key name, and mostly using
> no more than a couple of key version names.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13282:
Assignee: Steve Loughran
Status: Patch Available (was: Open)
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13282-001.patch
>
>
> If the etags of blobs were exported via {{getFileChecksum()}}, it'd be
> possible to probe for a blob being in sync with a local file. Distcp could
> use this to decide whether to skip a file or not.
> Now, there's a problem there: distcp needs source and dest filesystems to
> implement the same algorithm. It'd only work out the box if you were copying
> between S3 instances. There are also quirks with encryption and multipart:
> [s3
> docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
> At the very least, it's something which could be used when indexing the FS,
> to check for changes later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13282:
Attachment: HADOOP-13282-001.patch
Patch 001: code yes, tests no.
probably time to do a contract test which verifies that
* things fail on: getFileChecksum("/")
* as well as other directories
* two calls to the same file return the same checksum (whose bytes match)
* the same file, if overwritten with different data, has a different checksum
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13282-001.patch
>
>
> If the etags of blobs were exported via {{getFileChecksum()}}, it'd be
> possible to probe for a blob being in sync with a local file. Distcp could
> use this to decide whether to skip a file or not.
> Now, there's a problem there: distcp needs source and dest filesystems to
> implement the same algorithm. It'd only work out the box if you were copying
> between S3 instances. There are also quirks with encryption and multipart:
> [s3
> docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
> At the very least, it's something which could be used when indexing the FS,
> to check for changes later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14843) FsPermission symbolic parsing failed to detect invalid argument
[ https://issues.apache.org/jira/browse/HADOOP-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155948#comment-16155948 ] Jason Lowe commented on HADOOP-14843: - I believe HADOOP-13508 triggered the behavior change from interpreting the malformed argument as mode instead of mode 0777. However I wouldn't say this bug was caused by that JIRA. The FsPermission string constructor was accepting this malformed argument both before and after HADOOP-13508, and it should have thrown an error in either case. > FsPermission symbolic parsing failed to detect invalid argument > --- > > Key: HADOOP-14843 > URL: https://issues.apache.org/jira/browse/HADOOP-14843 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.4, 2.8.1 >Reporter: Jason Lowe >Assignee: Bharat Viswanadham > > A user misunderstood the syntax format for the FsPermission symbolic > constructor and passed the argument "-rwr" instead of "u=rw,g=r". In 2.7 and > earlier this was silently misinterpreted as mode 0777 and in 2.8 it oddly > became mode . In either case FsPermission should have flagged "-rwr" as > an invalid argument rather than silently misinterpreting it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155920#comment-16155920
]
Hadoop QA commented on HADOOP-14220:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 14
new + 21 unchanged - 0 fixed = 35 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
15s{color} | {color:red} hadoop-tools_hadoop-aws generated 2 new + 0 unchanged
- 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.getLongParam(Map,
String, long) At
DynamoDBMetadataStore.java:org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.getLongParam(Map,
String, long) At DynamoDBMetadataStore.java:[line 1099] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14220 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885652/HADOOP-14220-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b211be7dd9be 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f3bc63 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13180/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13180/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13180/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
|
| Test Results |
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155895#comment-16155895 ] Aaron Fabbri commented on HADOOP-14220: --- Thanks for the mention [~ste...@apache.org] I will review it now. > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155891#comment-16155891 ] Steve Loughran commented on HADOOP-14220: - I'd really like to get this reviewed and in to the beta, as it rounds off s3guard CLI with more functions, docs & tests. Any reviewers? [~fabbri]? [~raviprak]? > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14220: Affects Version/s: (was: HADOOP-13345) 3.0.0-beta1 Target Version/s: 3.0.0-beta1 (was: HADOOP-13345) > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14220: Attachment: HADOOP-14220-006.patch Patch 006 fix tabs to spaces & move from to ### in md where it'd break doxia > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[
https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155881#comment-16155881
]
Hadoop QA commented on HADOOP-14738:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
43s{color} | {color:green} root generated 0 new + 1282 unchanged - 1 fixed =
1282 total (was 1283) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 6s{color} | {color:orange} root: The patch generated 2 new + 23 unchanged -
103 fixed = 25 total (was 126) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 30 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-client-minicluster in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Comparison of String objects using
[jira] [Updated] (HADOOP-14103) Sort out hadoop-aws contract-test-options.xml
[ https://issues.apache.org/jira/browse/HADOOP-14103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14103: Fix Version/s: 3.1.0 > Sort out hadoop-aws contract-test-options.xml > - > > Key: HADOOP-14103 > URL: https://issues.apache.org/jira/browse/HADOOP-14103 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1, 3.1.0 > > Attachments: HADOOP-14103.001.patch, HADOOP-14103.002.patch, > HADOOP-14103.003.patch, HADOOP-14103.004.patch > > > The doc update of HADOOP-14099 has shown that there's confusion about whether > we need a src/test/resources/contract-test-options.xml file. > It's documented as needed, branch-2 has it in .gitignore; trunk doesn't. > I think it's needed for the contract tests, which the S3A test base extends > (And therefore needs). However, we can just put in an SCM managed one and > have it just XInclude auth-keys.xml > I propose: do that, fix up the testing docs to match -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14103) Sort out hadoop-aws contract-test-options.xml
[ https://issues.apache.org/jira/browse/HADOOP-14103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155852#comment-16155852 ] John Zhuge commented on HADOOP-14103: - Committed to branch-3.0 as well. > Sort out hadoop-aws contract-test-options.xml > - > > Key: HADOOP-14103 > URL: https://issues.apache.org/jira/browse/HADOOP-14103 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1, 3.1.0 > > Attachments: HADOOP-14103.001.patch, HADOOP-14103.002.patch, > HADOOP-14103.003.patch, HADOOP-14103.004.patch > > > The doc update of HADOOP-14099 has shown that there's confusion about whether > we need a src/test/resources/contract-test-options.xml file. > It's documented as needed, branch-2 has it in .gitignore; trunk doesn't. > I think it's needed for the contract tests, which the S3A test base extends > (And therefore needs). However, we can just put in an SCM managed one and > have it just XInclude auth-keys.xml > I propose: do that, fix up the testing docs to match -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14843) FsPermission symbolic parsing failed to detect invalid argument
[ https://issues.apache.org/jira/browse/HADOOP-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155785#comment-16155785 ] Wei-Chiu Chuang commented on HADOOP-14843: -- Is it caused by HADOOP-13508? HADOOP-13508 is/will be in 2.8.2 though > FsPermission symbolic parsing failed to detect invalid argument > --- > > Key: HADOOP-14843 > URL: https://issues.apache.org/jira/browse/HADOOP-14843 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.4, 2.8.1 >Reporter: Jason Lowe >Assignee: Bharat Viswanadham > > A user misunderstood the syntax format for the FsPermission symbolic > constructor and passed the argument "-rwr" instead of "u=rw,g=r". In 2.7 and > earlier this was silently misinterpreted as mode 0777 and in 2.8 it oddly > became mode . In either case FsPermission should have flagged "-rwr" as > an invalid argument rather than silently misinterpreting it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155769#comment-16155769
]
Hanisha Koneru commented on HADOOP-14827:
-
Thanks for the improvement, [~xkrogen].
The patch LGTM. TestZKFailoverController and TestShellBasedUnixGroupsMapping
pass locally for me too.
+1 (non-binding).
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155767#comment-16155767
]
Xiao Chen commented on HADOOP-14841:
Thanks Wei-Chiu for commenting.
bq. share the script and the reproduction steps?
(branch-2 kms) set {{KMS_MAX_THREADS}} to 1, try hitting it with curl commands
[for
decryption|http://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-kms/index.html#Decrypt_Encrypted_Key]
in multiple threads. (Can first run a generate, then use its output as the
decrypt input)
I remember seeing errors after a few minutes. (This was a while ago, so don't
have the scripts handy)
Given this was reproduce with curl, I think it slightly adds some credibility
it's the server that's at fault. Cannot think of a definitive way to point at
client/server side though. Should probably try again with jetty-based kms and
compare
More thoughts welcome. :)
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-14841.01.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
>
[jira] [Commented] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A
[
https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155746#comment-16155746
]
Aaron Fabbri commented on HADOOP-13421:
---
{quote}
better
fs.s3a.list.version="1"
line us up for a v3 algorithm in some time in the future.
{quote}
Ok.. Thought about that. What do you think behavior should be if the value is
out of range? 1. Fallback to default (v2) or 2. Fail to init S3A FS.
I'll roll another patch with that change and checkstyle fixes.
> Switch to v2 of the S3 List Objects API in S3A
> --
>
> Key: HADOOP-13421
> URL: https://issues.apache.org/jira/browse/HADOOP-13421
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steven K. Wong
>Assignee: Aaron Fabbri
>Priority: Minor
> Attachments: HADOOP-13421.002.patch,
> HADOOP-13421-HADOOP-13345.001.patch
>
>
> Unlike [version
> 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the
> S3 List Objects API, [version
> 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by
> default does not fetch object owner information, which S3A doesn't need
> anyway. By switching to v2, there will be less data to transfer/process.
> Also, it should be more robust when listing a versioned bucket with "a large
> number of delete markers" ([according to
> AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]).
> Methods in S3AFileSystem that use this API include:
> * getFileStatus(Path)
> * innerDelete(Path, boolean)
> * innerListStatus(Path)
> * innerRename(Path, Path)
> Requires AWS SDK 1.10.75 or later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12077) Provide a multi-URI replication Inode for ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155723#comment-16155723 ] Gera Shegalov commented on HADOOP-12077: Thanks [~chris.douglas] for taking care of fixing up the patch, and committing it > Provide a multi-URI replication Inode for ViewFs > > > Key: HADOOP-12077 > URL: https://issues.apache.org/jira/browse/HADOOP-12077 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-12077.001.patch, HADOOP-12077.002.patch, > HADOOP-12077.003.patch, HADOOP-12077.004.patch, HADOOP-12077.005.patch, > HADOOP-12077.006.patch, HADOOP-12077.007.patch, HADOOP-12077.008.patch, > HADOOP-12077.009.patch, HADOOP-12077.010.patch > > > This JIRA is to provide simple "replication" capabilities for applications > that maintain logically equivalent paths in multiple locations for caching or > failover (e.g., S3 and HDFS). We noticed a simple common HDFS usage pattern > in our applications. They host their data on some logical cluster C. There > are corresponding HDFS clusters in multiple datacenters. When the application > runs in DC1, it prefers to read from C in DC1, and the applications prefers > to failover to C in DC2 if the application is migrated to DC2 or when C in > DC1 is unavailable. New application data versions are created > periodically/relatively infrequently. > In order to address many common scenarios in a general fashion, and to avoid > unnecessary code duplication, we implement this functionality in ViewFs (our > default FileSystem spanning all clusters in all datacenters) in a project > code-named Nfly (N as in N datacenters). Currently each ViewFs Inode points > to a single URI via ChRootedFileSystem. Consequently, we introduce a new type > of links that points to a list of URIs that are each going to be wrapped in > ChRootedFileSystem. A typical usage: > /nfly/C/user->/DC1/C/user,/DC2/C/user,... This collection of > ChRootedFileSystem instances is fronted by the Nfly filesystem object that is > actually used for the mount point/Inode. Nfly filesystems backs a single > logical path /nfly/C/user//path by multiple physical paths. > Nfly filesystem supports setting minReplication. As long as the number of > URIs on which an update has succeeded is greater than or equal to > minReplication exceptions are only logged but not thrown. Each update > operation is currently executed serially (client-bandwidth driven parallelism > will be added later). > A file create/write: > # Creates a temporary invisible _nfly_tmp_file in the intended chrooted > filesystem. > # Returns a FSDataOutputStream that wraps output streams returned by 1 > # All writes are forwarded to each output stream. > # On close of stream created by 2, all n streams are closed, and the files > are renamed from _nfly_tmp_file to file. All files receive the same mtime > corresponding to the client system time as of beginning of this step. > # If at least minReplication destinations has gone through steps 1-4 without > failures the transaction is considered logically committed, otherwise a > best-effort attempt of cleaning up the temporary files is attempted. > As for reads, we support a notion of locality similar to HDFS /DC/rack/node. > We sort Inode URIs using NetworkTopology by their authorities. These are > typically host names in simple HDFS URIs. If the authority is missing as is > the case with the local file:/// the local host name is assumed > InetAddress.getLocalHost(). This makes sure that the local file system is > always the closest one to the reader in this approach. For our Hadoop 2 hdfs > URIs that are based on nameservice ids instead of hostnames it is very easy > to adjust the topology script since our nameservice ids already contain the > datacenter. As for rack and node we can simply output any string such as > /DC/rack-nsid/node-nsid, since we only care about datacenter-locality for > such filesystem clients. > There are 2 policies/additions to the read call path that makes it more > expensive, but improve user experience: > - readMostRecent - when this policy is enabled, Nfly first checks mtime for > the path under all URIs, sorts them from most recent to least recent. Nfly > then sorts the set of most recent URIs topologically in the same manner as > described above. > - repairOnRead - when readMostRecent is enabled Nfly already has to RPC all > underlying destinations. With repairOnRead, Nfly filesystem would > additionally attempt to refresh destinations with the path missing or a stale > version of the path using the nearest available most recent destination. -- This message was
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155709#comment-16155709
]
Wei-Chiu Chuang commented on HADOOP-14841:
--
bq. Intentionally reproducing this on a KMS is pretty difficult, but I was able
to reproduce it if by setting kms max threads to 1 and script-generating some
loads (with the old tomcat version).
Hi [~xiaochen] would you like to also share the script and the reproduction
steps? I share the same concern with [~shahrs87] that it wasn't clear whether
it is a client-side or server-side bug. That said, the same error can be seen
when you send a POST request to KMS with _empty_ body. (A POST request with
malformed body has a different error)
[~shahrs87] to give a bit more context, we saw this error during stress tests
and on some busy clusters, and the error seems to go away when we added more
KMS instances to the cluster for load balancing.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-14841.01.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
>
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155705#comment-16155705
]
Xiao Chen commented on HADOOP-14841:
Thanks for looking [~shahrs87].
bq. it doesn't look like client side bug but without client side logs not able
to confirm anything.
I'm on the same lines here. Checked client code carefully but didn't see
anywhere this could be a problem.
Client log doesn't show anything before the exception. The major difficulty of
this issue is reproducing it, though we have see several production occurrences
that goes away on the next run, and this only happens when the kms jmx shows a
significant load increase.
bq. How are we sure that writer is fine ?
This was explained in the next part of that comment, due to different
requirements. Moreover, for all the occurrences it's always the reader that
throws.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-14841.01.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
>
[jira] [Assigned] (HADOOP-14843) FsPermission symbolic parsing failed to detect invalid argument
[ https://issues.apache.org/jira/browse/HADOOP-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HADOOP-14843: --- Assignee: Bharat Viswanadham > FsPermission symbolic parsing failed to detect invalid argument > --- > > Key: HADOOP-14843 > URL: https://issues.apache.org/jira/browse/HADOOP-14843 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.4, 2.8.1 >Reporter: Jason Lowe >Assignee: Bharat Viswanadham > > A user misunderstood the syntax format for the FsPermission symbolic > constructor and passed the argument "-rwr" instead of "u=rw,g=r". In 2.7 and > earlier this was silently misinterpreted as mode 0777 and in 2.8 it oddly > became mode . In either case FsPermission should have flagged "-rwr" as > an invalid argument rather than silently misinterpreting it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14738: Status: Patch Available (was: Open) this patch is targeting 3.0 only, BTW. We could have 2.9 do the earlier warning-of-pending-doom log option & leave the docs alone there > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs > -- > > Key: HADOOP-14738 > URL: https://issues.apache.org/jira/browse/HADOOP-14738 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-14738-002.patch, HADOOP-14738-003.patch, > HADOOP-14739-001.patch > > > We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf > since Hadoop 2.8 > It's now time to kill S3N off, remove the source, the tests, the transitive > dependencies. > I propose that in Hadoop 3.0 beta we tell people off from using it, and link > to a doc page (wiki?) about how to migrate (Change URLs, update config ops). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14738: Attachment: HADOOP-14738-003.patch Patch 003 h2. S3N * remove all traces of s3n throughout the project, XML, docs, resources, test-resources * NativeS3FileSystem now a stub class which throws an IOE in its initialize call. * simplified other bits of docs (e.g. testing.md) where there's now less to worry about * Jets3t from all poms h2. docs * add root index.md.vm entry as requested * fix site generation (DOXIA-533) by removing all level4 headers * encryption doc added; mix of HDP content and some other bits * more subsections in troubleshooting * review all docs, site generation, ... > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs > -- > > Key: HADOOP-14738 > URL: https://issues.apache.org/jira/browse/HADOOP-14738 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-14738-002.patch, HADOOP-14738-003.patch, > HADOOP-14739-001.patch > > > We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf > since Hadoop 2.8 > It's now time to kill S3N off, remove the source, the tests, the transitive > dependencies. > I propose that in Hadoop 3.0 beta we tell people off from using it, and link > to a doc page (wiki?) about how to migrate (Change URLs, update config ops). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155676#comment-16155676
]
Rushabh S Shah commented on HADOOP-14841:
-
[~xiaochen]: By any chance do you have client side stack trace or log ?
bq. Why the input stream is empty? Not 100% sure in our case, but very likely
3rd party...
I think we are not sure what caused the EOFException. It _can be_ malformed
jsonpayload from the client.
Quickly looking at the client side code, it doesn't look like client side bug
but without client side logs not able to confirm anything.
bq. The exception comes from KMSJSONReader, when it tries to deserialize the
json object. Writer seems fine.
How are we sure that writer is fine ?
bq. The retry policy itself is pretty widely used, so perhaps we can throw a
RetriableException from KMSJSONReader to make the KMSCP retry.
Without understanding the root cause, I am little bit hesitant to retry on
EOFException.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-14841.01.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
>
[jira] [Resolved] (HADOOP-13894) s3a troubleshooting to cover the "JSON parse error" message
[ https://issues.apache.org/jira/browse/HADOOP-13894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13894. - Resolution: Duplicate > s3a troubleshooting to cover the "JSON parse error" message > --- > > Key: HADOOP-13894 > URL: https://issues.apache.org/jira/browse/HADOOP-13894 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Priority: Minor > > Generally problems in s3 IO during list operations surface as JSON parse > errors, with the underlying cause lost (unchecked HTTP error code, > text/plain, text/html, interrupted thread). > Document this fact in the troubleshooting section -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14835) mvn site build throws SAX errors
[
https://issues.apache.org/jira/browse/HADOOP-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1614#comment-1614
]
Allen Wittenauer edited comment on HADOOP-14835 at 9/6/17 3:53 PM:
---
Ripping out YARN-6877, javadoc appears to be working but I'm failing in
hadoop-client-check-test-invariants with
{code}
WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-runtime:jar:3.1.0-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-minicluster:jar:3.1.0-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/org/apache/xerces/impl/dv/dtd/NMTOKENDatatypeValidator.class
{code}
was (Author: aw):
Ripping out YARN-6877, I'm still failing in hadoop-client-check-test-invariants
with
{code}
WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-runtime:jar:3.1.0-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-minicluster:jar:3.1.0-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/org/apache/xerces/impl/dv/dtd/NMTOKENDatatypeValidator.class
{code}
> mvn site build throws SAX errors
>
>
> Key: HADOOP-14835
> URL: https://issues.apache.org/jira/browse/HADOOP-14835
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, site
>Affects Versions: 3.0.0-beta1
>Reporter: Allen Wittenauer
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: HADOOP-14835.001.patch
>
>
> Running mvn install site site:stage -DskipTests -Pdist,src
> -Preleasedocs,docs results in a stack trace when run on a fresh .m2
> directory. It appears to be coming from the jdiff doclets in the annotations
> code.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14835) mvn site build throws SAX errors
[
https://issues.apache.org/jira/browse/HADOOP-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1614#comment-1614
]
Allen Wittenauer commented on HADOOP-14835:
---
Ripping out YARN-6877, I'm still failing in hadoop-client-check-test-invariants
with
{code}
WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-runtime:jar:3.1.0-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-minicluster:jar:3.1.0-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/org/apache/xerces/impl/dv/dtd/NMTOKENDatatypeValidator.class
{code}
> mvn site build throws SAX errors
>
>
> Key: HADOOP-14835
> URL: https://issues.apache.org/jira/browse/HADOOP-14835
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, site
>Affects Versions: 3.0.0-beta1
>Reporter: Allen Wittenauer
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: HADOOP-14835.001.patch
>
>
> Running mvn install site site:stage -DskipTests -Pdist,src
> -Preleasedocs,docs results in a stack trace when run on a fresh .m2
> directory. It appears to be coming from the jdiff doclets in the annotations
> code.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14103) Sort out hadoop-aws contract-test-options.xml
[ https://issues.apache.org/jira/browse/HADOOP-14103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155539#comment-16155539 ] John Zhuge commented on HADOOP-14103: - Ok, I will not backport to 2.8 then. > Sort out hadoop-aws contract-test-options.xml > - > > Key: HADOOP-14103 > URL: https://issues.apache.org/jira/browse/HADOOP-14103 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14103.001.patch, HADOOP-14103.002.patch, > HADOOP-14103.003.patch, HADOOP-14103.004.patch > > > The doc update of HADOOP-14099 has shown that there's confusion about whether > we need a src/test/resources/contract-test-options.xml file. > It's documented as needed, branch-2 has it in .gitignore; trunk doesn't. > I think it's needed for the contract tests, which the S3A test base extends > (And therefore needs). However, we can just put in an SCM managed one and > have it just XInclude auth-keys.xml > I propose: do that, fix up the testing docs to match -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155495#comment-16155495 ] ASF GitHub Bot commented on HADOOP-14217: - Github user yufeldman commented on the issue: https://github.com/apache/hadoop/pull/269 will do > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/oss >Affects Versions: 2.8.1 >Reporter: Genmao Yu >Assignee: Yuliya Feldman > Attachments: Colon handling in hadoop Path.pdf > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14364) refresh changelog/release notes with newer Apache Yetus build
[ https://issues.apache.org/jira/browse/HADOOP-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155423#comment-16155423 ] Steve Loughran commented on HADOOP-14364: - I'm a bit concerned that this has added a -SNAPSHOT import to the build > refresh changelog/release notes with newer Apache Yetus build > - > > Key: HADOOP-14364 > URL: https://issues.apache.org/jira/browse/HADOOP-14364 > Project: Hadoop Common > Issue Type: Bug > Components: build, documentation >Affects Versions: 3.0.0-alpha4 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14364.00.patch, HADOOP-14364.01.patch, > HADOOP-14364.02.patch, HADOOP-14364.03.patch, HADOOP-14364.04.patch > > > A lot of fixes went into Apache Yetus 0.4.0 wrt releasedocs and how it's > output gets rendered with mvn site. We should re-run releasedocs for all > releases and refresh the content to use the new formatting. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14738: Attachment: HADOOP-14738-002.patch HADOOP-14738 002: reinstate jets3t; add index.vm ref, and more anchors in the aws sections This patch still has the s3n binaries & tests, so can be applied to branch-2 . I'm going to do another iteration with s3n pulled other than a stub class; tests will be cut along with config details > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs > -- > > Key: HADOOP-14738 > URL: https://issues.apache.org/jira/browse/HADOOP-14738 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-14738-002.patch, HADOOP-14739-001.patch > > > We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf > since Hadoop 2.8 > It's now time to kill S3N off, remove the source, the tests, the transitive > dependencies. > I propose that in Hadoop 3.0 beta we tell people off from using it, and link > to a doc page (wiki?) about how to migrate (Change URLs, update config ops). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14843) FsPermission symbolic parsing failed to detect invalid argument
Jason Lowe created HADOOP-14843: --- Summary: FsPermission symbolic parsing failed to detect invalid argument Key: HADOOP-14843 URL: https://issues.apache.org/jira/browse/HADOOP-14843 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.8.1, 2.7.4 Reporter: Jason Lowe A user misunderstood the syntax format for the FsPermission symbolic constructor and passed the argument "-rwr" instead of "u=rw,g=r". In 2.7 and earlier this was silently misinterpreted as mode 0777 and in 2.8 it oddly became mode . In either case FsPermission should have flagged "-rwr" as an invalid argument rather than silently misinterpreting it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14738: Status: Open (was: Patch Available) > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs > -- > > Key: HADOOP-14738 > URL: https://issues.apache.org/jira/browse/HADOOP-14738 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-14739-001.patch > > > We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf > since Hadoop 2.8 > It's now time to kill S3N off, remove the source, the tests, the transitive > dependencies. > I propose that in Hadoop 3.0 beta we tell people off from using it, and link > to a doc page (wiki?) about how to migrate (Change URLs, update config ops). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155327#comment-16155327
]
Hadoop QA commented on HADOOP-14839:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 44s{color}
| {color:red} hadoop-distcp in the patch failed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
13s{color} | {color:green} hadoop-extras in the patch passed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests | hadoop.tools.TestIntegration |
| | hadoop.tools.TestDistCpViewFs |
| JDK v1.7.0_131 Failed junit tests | hadoop.tools.TestIntegration |
| | hadoop.tools.TestDistCpViewFs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA Issue | HADOOP-14839 |
| JIRA Patch URL |
[jira] [Commented] (HADOOP-13578) Add Codec for ZStandard Compression
[ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155295#comment-16155295 ] Andre F de Miranda commented on HADOOP-13578: - [~churromorales] was this ever backported to 2.7.x? I couldn't find the mentioned JIRA Cheers > Add Codec for ZStandard Compression > --- > > Key: HADOOP-13578 > URL: https://issues.apache.org/jira/browse/HADOOP-13578 > Project: Hadoop Common > Issue Type: New Feature >Reporter: churro morales >Assignee: churro morales > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HADOOP-13578-branch-2.v9.patch, HADOOP-13578.patch, > HADOOP-13578.v1.patch, HADOOP-13578.v2.patch, HADOOP-13578.v3.patch, > HADOOP-13578.v4.patch, HADOOP-13578.v5.patch, HADOOP-13578.v6.patch, > HADOOP-13578.v7.patch, HADOOP-13578.v8.patch, HADOOP-13578.v9.patch > > > ZStandard: https://github.com/facebook/zstd has been used in production for 6 > months by facebook now. v1.0 was recently released. Create a codec for this > library. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HADOOP-14839:
---
Attachment: HADOOP-14839-branch-2.002.patch
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HADOOP-14839-branch-2.001.patch,
> HADOOP-14839-branch-2.002.patch, HDFS-10234.001.patch, HDFS-10234.002.patch,
> HDFS-10234.003.patch, HDFS-10234.004.patch, HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155276#comment-16155276
]
Yiqun Lin commented on HADOOP-14839:
The failure test {{TestOptionsParser}} is related and other two can be passed
in my local.
Attach the updated patch of branch-2.
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HADOOP-14839-branch-2.001.patch,
> HDFS-10234.001.patch, HDFS-10234.002.patch, HDFS-10234.003.patch,
> HDFS-10234.004.patch, HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A
[
https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155261#comment-16155261
]
Steve Loughran commented on HADOOP-13421:
-
{code}
fs.s3a.use.list.v1
{code}
better
{code}
fs.s3a.list.version="1"
{code}
line us up for a v3 algorithm in some time in the future. This will propagate
into the s3a FS code too.
* Can just call the class {{ListRequest}}, or, if you really want an s3 prefix.
One thing to consider: what if someone who only has the v1 API wants to run the
tests? I know Thomas and Ewan are happy, but don't know about others. Maybe: if
the version API is explicitly set to v1 for a bucket then skip the V2 tests?
Looking at the code though, I think that's a moot issue. If someone forces the
list API to v1 in their settings, that will be implicitly picked yp by
everything except the v1 test, which just becomes a duplicate of the superclass
in terms of test coverage...its not going to fail.
> Switch to v2 of the S3 List Objects API in S3A
> --
>
> Key: HADOOP-13421
> URL: https://issues.apache.org/jira/browse/HADOOP-13421
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steven K. Wong
>Assignee: Aaron Fabbri
>Priority: Minor
> Attachments: HADOOP-13421.002.patch,
> HADOOP-13421-HADOOP-13345.001.patch
>
>
> Unlike [version
> 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the
> S3 List Objects API, [version
> 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by
> default does not fetch object owner information, which S3A doesn't need
> anyway. By switching to v2, there will be less data to transfer/process.
> Also, it should be more robust when listing a versioned bucket with "a large
> number of delete markers" ([according to
> AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]).
> Methods in S3AFileSystem that use this API include:
> * getFileStatus(Path)
> * innerDelete(Path, boolean)
> * innerListStatus(Path)
> * innerRename(Path, Path)
> Requires AWS SDK 1.10.75 or later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14103) Sort out hadoop-aws contract-test-options.xml
[ https://issues.apache.org/jira/browse/HADOOP-14103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155243#comment-16155243 ] Steve Loughran commented on HADOOP-14103: - aah, don't worry about that > Sort out hadoop-aws contract-test-options.xml > - > > Key: HADOOP-14103 > URL: https://issues.apache.org/jira/browse/HADOOP-14103 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14103.001.patch, HADOOP-14103.002.patch, > HADOOP-14103.003.patch, HADOOP-14103.004.patch > > > The doc update of HADOOP-14099 has shown that there's confusion about whether > we need a src/test/resources/contract-test-options.xml file. > It's documented as needed, branch-2 has it in .gitignore; trunk doesn't. > I think it's needed for the contract tests, which the S3A test base extends > (And therefore needs). However, we can just put in an SCM managed one and > have it just XInclude auth-keys.xml > I propose: do that, fix up the testing docs to match -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155230#comment-16155230 ] Steve Loughran commented on HADOOP-14738: - A full cut of s3n? I'm happy with that. in which case I'd just have a stub child FS which declared itself as missing & switch the s3 docs to a "how to migrate" doc alone > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1; rework docs > -- > > Key: HADOOP-14738 > URL: https://issues.apache.org/jira/browse/HADOOP-14738 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-14739-001.patch > > > We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf > since Hadoop 2.8 > It's now time to kill S3N off, remove the source, the tests, the transitive > dependencies. > I propose that in Hadoop 3.0 beta we tell people off from using it, and link > to a doc page (wiki?) about how to migrate (Change URLs, update config ops). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155222#comment-16155222 ] ASF GitHub Bot commented on HADOOP-14217: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/269 I think some new Abstract FS Contract test will be needed to see how filesystems actually handle ":" chars; having the path support it is just one part of the problem > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/oss >Affects Versions: 2.8.1 >Reporter: Genmao Yu >Assignee: Yuliya Feldman > Attachments: Colon handling in hadoop Path.pdf > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155163#comment-16155163
]
Hadoop QA commented on HADOOP-14839:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-tools: The patch generated 1 new + 211
unchanged - 0 fixed = 212 total (was 211) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 58s{color}
| {color:red} hadoop-distcp in the patch failed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
6s{color} | {color:green} hadoop-extras in the patch passed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests | hadoop.tools.TestOptionsParser |
| | hadoop.tools.TestIntegration |
| | hadoop.tools.TestDistCpViewFs |
| JDK v1.7.0_131 Failed junit tests | hadoop.tools.TestOptionsParser |
| | hadoop.tools.TestIntegration |
| | hadoop.tools.TestDistCpViewFs |
\\
\\
||
[jira] [Commented] (HADOOP-14808) Hadoop keychain
[
https://issues.apache.org/jira/browse/HADOOP-14808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155077#comment-16155077
]
Hadoop QA commented on HADOOP-14808:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 6s{color} | {color:orange} root: The patch generated 6 new + 354 unchanged
- 3 fixed = 360 total (was 357) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} hadoop-tools/hadoop-azure-datalake generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 23s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-azure-datalake |
| | Found reliance on default encoding in
org.apache.hadoop.fs.adl.AzureCLIKeychainLoader.loadAccessTokensFile(Credentials,
File):in
org.apache.hadoop.fs.adl.AzureCLIKeychainLoader.loadAccessTokensFile(Credentials,
File): String.getBytes() At AzureCLIKeychainLoader.java:[line 65] |
| Failed junit tests | hadoop.net.TestDNS |
| | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14808 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885543/HADOOP-14808.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 6a7ee11b012a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
[jira] [Updated] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HADOOP-14839:
---
Attachment: HADOOP-14839-branch-2.001.patch
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HADOOP-14839-branch-2.001.patch,
> HDFS-10234.001.patch, HDFS-10234.002.patch, HDFS-10234.003.patch,
> HDFS-10234.004.patch, HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin reopened HADOOP-14839:
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HADOOP-14839-branch-2.001.patch,
> HDFS-10234.001.patch, HDFS-10234.002.patch, HDFS-10234.003.patch,
> HDFS-10234.004.patch, HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HADOOP-14839:
---
Status: Patch Available (was: Reopened)
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HADOOP-14839-branch-2.001.patch,
> HDFS-10234.001.patch, HDFS-10234.002.patch, HDFS-10234.003.patch,
> HDFS-10234.004.patch, HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155065#comment-16155065
]
Yiqun Lin commented on HADOOP-14839:
Thanks Xiaoyu, attach the patch for branch-2.
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HDFS-10234.001.patch,
> HDFS-10234.002.patch, HDFS-10234.003.patch, HDFS-10234.004.patch,
> HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14103) Sort out hadoop-aws contract-test-options.xml
[ https://issues.apache.org/jira/browse/HADOOP-14103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14103: Resolution: Fixed Fix Version/s: 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks [~steve_l] for the review! It is tricky to backport the patch to branch-2.8. Still working on it. > Sort out hadoop-aws contract-test-options.xml > - > > Key: HADOOP-14103 > URL: https://issues.apache.org/jira/browse/HADOOP-14103 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14103.001.patch, HADOOP-14103.002.patch, > HADOOP-14103.003.patch, HADOOP-14103.004.patch > > > The doc update of HADOOP-14099 has shown that there's confusion about whether > we need a src/test/resources/contract-test-options.xml file. > It's documented as needed, branch-2 has it in .gitignore; trunk doesn't. > I think it's needed for the contract tests, which the S3A test base extends > (And therefore needs). However, we can just put in an SCM managed one and > have it just XInclude auth-keys.xml > I propose: do that, fix up the testing docs to match -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14808) Hadoop keychain
[
https://issues.apache.org/jira/browse/HADOOP-14808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14808:
Attachment: HADOOP-14808.002.patch
Patch 002
* Fix checkstyle and findbugs
* Fix TestCredentialProviderFactory failure
> Hadoop keychain
> ---
>
> Key: HADOOP-14808
> URL: https://issues.apache.org/jira/browse/HADOOP-14808
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.7.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14808.001.patch, HADOOP-14808.002.patch
>
>
> Extend the idea from HADOOP-6520 "UGI should load tokens from the
> environment" to a generic lightweight "keychain" design. Load keys (secrets)
> into a keychain in UGI (secret map) at startup. YARN will distribute them
> securely into each container. The Hadoop code running in the container can
> then retrieve the credentials from UGI.
> The use case is Bring Your Own Key (BYOK) credentials for cloud connectors
> (adl, wasb, s3a, etc.), while Hadoop authentication is still Kerberos. No
> configuration change, no admin involved. It will support YARN applications
> initially, e.g., DistCp, Tera Suite, Spark-on-Yarn, etc.
> Implementation is surprisingly simple because almost all pieces are in place:
> * Retrieve secrets from UGI using {{conf.getPassword}} backed by the existing
> Credential Provider class {{UserProvider}}
> * Reuse Credential Provider classes and interface to define local permanent
> or transient credential store, e.g., {{LocalJavaKeyStoreProvider}}
> * New: create a new transient Credential Provider that logs into AAD with
> username/password or device code, and then put the Client ID and Refresh
> Token into the keychain
> * New: create a new permanent Credential Provider based on Hadoop
> configuration XML, for dev/testing purpose.
> Links
> * HADOOP-11766 Generic token authentication support for Hadoop
> * HADOOP-11744 Support OAuth2 in Hadoop
> * HADOOP-10959 A Kerberos based token authentication approach
> * HADOOP-9392 Token based authentication and Single Sign On
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12077) Provide a multi-URI replication Inode for ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-12077: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 Status: Resolved (was: Patch Available) +1 I committed this. Thanks Gera > Provide a multi-URI replication Inode for ViewFs > > > Key: HADOOP-12077 > URL: https://issues.apache.org/jira/browse/HADOOP-12077 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-12077.001.patch, HADOOP-12077.002.patch, > HADOOP-12077.003.patch, HADOOP-12077.004.patch, HADOOP-12077.005.patch, > HADOOP-12077.006.patch, HADOOP-12077.007.patch, HADOOP-12077.008.patch, > HADOOP-12077.009.patch, HADOOP-12077.010.patch > > > This JIRA is to provide simple "replication" capabilities for applications > that maintain logically equivalent paths in multiple locations for caching or > failover (e.g., S3 and HDFS). We noticed a simple common HDFS usage pattern > in our applications. They host their data on some logical cluster C. There > are corresponding HDFS clusters in multiple datacenters. When the application > runs in DC1, it prefers to read from C in DC1, and the applications prefers > to failover to C in DC2 if the application is migrated to DC2 or when C in > DC1 is unavailable. New application data versions are created > periodically/relatively infrequently. > In order to address many common scenarios in a general fashion, and to avoid > unnecessary code duplication, we implement this functionality in ViewFs (our > default FileSystem spanning all clusters in all datacenters) in a project > code-named Nfly (N as in N datacenters). Currently each ViewFs Inode points > to a single URI via ChRootedFileSystem. Consequently, we introduce a new type > of links that points to a list of URIs that are each going to be wrapped in > ChRootedFileSystem. A typical usage: > /nfly/C/user->/DC1/C/user,/DC2/C/user,... This collection of > ChRootedFileSystem instances is fronted by the Nfly filesystem object that is > actually used for the mount point/Inode. Nfly filesystems backs a single > logical path /nfly/C/user//path by multiple physical paths. > Nfly filesystem supports setting minReplication. As long as the number of > URIs on which an update has succeeded is greater than or equal to > minReplication exceptions are only logged but not thrown. Each update > operation is currently executed serially (client-bandwidth driven parallelism > will be added later). > A file create/write: > # Creates a temporary invisible _nfly_tmp_file in the intended chrooted > filesystem. > # Returns a FSDataOutputStream that wraps output streams returned by 1 > # All writes are forwarded to each output stream. > # On close of stream created by 2, all n streams are closed, and the files > are renamed from _nfly_tmp_file to file. All files receive the same mtime > corresponding to the client system time as of beginning of this step. > # If at least minReplication destinations has gone through steps 1-4 without > failures the transaction is considered logically committed, otherwise a > best-effort attempt of cleaning up the temporary files is attempted. > As for reads, we support a notion of locality similar to HDFS /DC/rack/node. > We sort Inode URIs using NetworkTopology by their authorities. These are > typically host names in simple HDFS URIs. If the authority is missing as is > the case with the local file:/// the local host name is assumed > InetAddress.getLocalHost(). This makes sure that the local file system is > always the closest one to the reader in this approach. For our Hadoop 2 hdfs > URIs that are based on nameservice ids instead of hostnames it is very easy > to adjust the topology script since our nameservice ids already contain the > datacenter. As for rack and node we can simply output any string such as > /DC/rack-nsid/node-nsid, since we only care about datacenter-locality for > such filesystem clients. > There are 2 policies/additions to the read call path that makes it more > expensive, but improve user experience: > - readMostRecent - when this policy is enabled, Nfly first checks mtime for > the path under all URIs, sorts them from most recent to least recent. Nfly > then sorts the set of most recent URIs topologically in the same manner as > described above. > - repairOnRead - when readMostRecent is enabled Nfly already has to RPC all > underlying destinations. With repairOnRead, Nfly filesystem would > additionally attempt to refresh destinations with the path missing or a stale > version of the path using the nearest available
[jira] [Commented] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154884#comment-16154884
]
Xiaoyu Yao commented on HADOOP-14839:
-
[~linyiqun], can you submit a patch for branch-2 as this one can be a useful
patch to back port?
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HDFS-10234.001.patch,
> HDFS-10234.002.patch, HDFS-10234.003.patch, HDFS-10234.004.patch,
> HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-14839:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
Thanks [~linyiqun] for the contribution and all for the reviews and
discussions. I've commit the patch to trunk and branch-3.0.
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HDFS-10234.001.patch,
> HDFS-10234.002.patch, HDFS-10234.003.patch, HDFS-10234.004.patch,
> HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
