[jira] [Commented] (HADOOP-15175) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[
https://issues.apache.org/jira/browse/HADOOP-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328406#comment-16328406
]
genericqa commented on HADOOP-15175:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 212 unchanged - 0 fixed = 215 total (was 212) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15175 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893880/HDFS-12532.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c946a1603488 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 41049ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13980/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13980/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13980/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1347 (vs.
[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A
[
https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328389#comment-16328389
]
genericqa commented on HADOOP-15141:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 57s{color}
| {color:red} root generated 2 new + 1239 unchanged - 0 fixed = 1241 total (was
1239) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 15s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
40s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15141 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906339/HADOOP-15141-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux d126df0eda4a 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 41049ba |
|
[jira] [Updated] (HADOOP-15167) [viewfs] ls will fail when user doesn't exist
[
https://issues.apache.org/jira/browse/HADOOP-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-15167:
--
Attachment: HADOOP-15167-002.patch
> [viewfs] ls will fail when user doesn't exist
> -
>
> Key: HADOOP-15167
> URL: https://issues.apache.org/jira/browse/HADOOP-15167
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15167-002.patch, HADOOP-15167.patch
>
>
> Have Secure federated cluster with atleast two nameservices
> Configure viewfs related configs
> When we run the {{ls}} cmd in HDFS client ,we will call the method:
> org.apache.hadoop.fs.viewfs.ViewFileSystem.InternalDirOfViewFs#getFileStatus
> it will try to get the group of the kerberos user. If the node has not this
> user, it fails.
> Throws the following and exits.UserGroupInformation#getPrimaryGroupName
> {code}
> if (groups.isEmpty()) {
> throw new IOException("There is no primary group for UGI " + this);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15142) Register FTP and SFTP as FS services
[
https://issues.apache.org/jira/browse/HADOOP-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328372#comment-16328372
]
genericqa commented on HADOOP-15142:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 43s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
35s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15142 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906338/HADOOP-15142.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 6908b1dd23f9 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 41049ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13978/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace
[jira] [Moved] (HADOOP-15175) DN Reg can Fail when principal doesn't contain hostname and floatingIP is configured.
[ https://issues.apache.org/jira/browse/HADOOP-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula moved HDFS-12532 to HADOOP-15175: -- Key: HADOOP-15175 (was: HDFS-12532) Project: Hadoop Common (was: Hadoop HDFS) > DN Reg can Fail when principal doesn't contain hostname and floatingIP is > configured. > - > > Key: HADOOP-15175 > URL: https://issues.apache.org/jira/browse/HADOOP-15175 > Project: Hadoop Common > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12532.patch > > > Configure principal without hostname (i.e hdfs/had...@hadoop.com) > Configure floatingIP > Start Cluster. > Here DN will fail to register as it can take IP which is not in "/etc/hosts". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328302#comment-16328302
]
genericqa commented on HADOOP-15027:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 43m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15027 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906198/HADOOP-15027.014.patch
|
| Optional Tests | asflicense findbugs xml compile javac javadoc
mvninstall mvnsite unit shadedclient checkstyle |
| uname | Linux 94bedbe053fb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 41049ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13977/testReport/ |
| Max. process+thread count | 439 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13977/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: Support
[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A
[
https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328290#comment-16328290
]
Aaron Fabbri commented on HADOOP-15141:
---
v6 patch: just fixing checkstyle and whitespace. Tested in us-west-2 w/ local
S3Guard. Will test w/ dynamodb while yetus runs here.
> Support IAM Assumed roles in S3A
>
>
> Key: HADOOP-15141
> URL: https://issues.apache.org/jira/browse/HADOOP-15141
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15141-001.patch, HADOOP-15141-002.patch,
> HADOOP-15141-003.patch, HADOOP-15141-004.patch, HADOOP-15141-005.patch,
> HADOOP-15141-006.patch
>
>
> Add the ability to use assumed roles in S3A
> * Add a property fs.s3a.assumed.role.arn for the ARN of the assumed role
> * add a new provider which grabs that and other properties and then creates a
> {{STSAssumeRoleSessionCredentialsProvider}} from it.
> * This also needs to support building up its own list of aws credential
> providers, from a different property; make the changes to S3AUtils for that
> * Tests
> * docs
> * and have the AwsProviderList forward closeable to it.
> * Get picked up automatically by DDB/s3guard
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15141) Support IAM Assumed roles in S3A
[
https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-15141:
--
Attachment: HADOOP-15141-006.patch
> Support IAM Assumed roles in S3A
>
>
> Key: HADOOP-15141
> URL: https://issues.apache.org/jira/browse/HADOOP-15141
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15141-001.patch, HADOOP-15141-002.patch,
> HADOOP-15141-003.patch, HADOOP-15141-004.patch, HADOOP-15141-005.patch,
> HADOOP-15141-006.patch
>
>
> Add the ability to use assumed roles in S3A
> * Add a property fs.s3a.assumed.role.arn for the ARN of the assumed role
> * add a new provider which grabs that and other properties and then creates a
> {{STSAssumeRoleSessionCredentialsProvider}} from it.
> * This also needs to support building up its own list of aws credential
> providers, from a different property; make the changes to S3AUtils for that
> * Tests
> * docs
> * and have the AwsProviderList forward closeable to it.
> * Get picked up automatically by DDB/s3guard
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328288#comment-16328288
]
genericqa commented on HADOOP-15027:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15027 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906198/HADOOP-15027.014.patch
|
| Optional Tests | asflicense findbugs xml compile javac javadoc
mvninstall mvnsite unit shadedclient checkstyle |
| uname | Linux ffb617251e00 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 41049ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13976/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13976/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: Support
[jira] [Commented] (HADOOP-15142) Register FTP and SFTP as FS services
[ https://issues.apache.org/jira/browse/HADOOP-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328284#comment-16328284 ] Mario Molina commented on HADOOP-15142: --- [~ste...@apache.org], I just submitted the patch, is that ok? > Register FTP and SFTP as FS services > > > Key: HADOOP-15142 > URL: https://issues.apache.org/jira/browse/HADOOP-15142 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.0.0 >Reporter: Mario Molina >Priority: Minor > Fix For: 3.0.1 > > Attachments: HADOOP-15142.001.patch, HADOOP-15142.002.patch > > > SFTPFileSystem and FTPFileSystem are not registered as a FS services. > When calling the 'get' or 'newInstance' methods of the FileSystem class, the > FS instance cannot be created due to the schema is not registered as a > service FS. > Also, the SFTPFileSystem class doesn't have the getScheme method implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15142) Register FTP and SFTP as FS services
[ https://issues.apache.org/jira/browse/HADOOP-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mario Molina updated HADOOP-15142: -- Attachment: HADOOP-15142.002.patch > Register FTP and SFTP as FS services > > > Key: HADOOP-15142 > URL: https://issues.apache.org/jira/browse/HADOOP-15142 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.0.0 >Reporter: Mario Molina >Priority: Minor > Fix For: 3.0.1 > > Attachments: HADOOP-15142.001.patch, HADOOP-15142.002.patch > > > SFTPFileSystem and FTPFileSystem are not registered as a FS services. > When calling the 'get' or 'newInstance' methods of the FileSystem class, the > FS instance cannot be created due to the schema is not registered as a > service FS. > Also, the SFTPFileSystem class doesn't have the getScheme method implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15142) Register FTP and SFTP as FS services
[ https://issues.apache.org/jira/browse/HADOOP-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mario Molina updated HADOOP-15142: -- Attachment: hadoop-15142.002.patch > Register FTP and SFTP as FS services > > > Key: HADOOP-15142 > URL: https://issues.apache.org/jira/browse/HADOOP-15142 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.0.0 >Reporter: Mario Molina >Priority: Minor > Fix For: 3.0.1 > > Attachments: HADOOP-15142.001.patch > > > SFTPFileSystem and FTPFileSystem are not registered as a FS services. > When calling the 'get' or 'newInstance' methods of the FileSystem class, the > FS instance cannot be created due to the schema is not registered as a > service FS. > Also, the SFTPFileSystem class doesn't have the getScheme method implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15142) Register FTP and SFTP as FS services
[ https://issues.apache.org/jira/browse/HADOOP-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mario Molina updated HADOOP-15142: -- Attachment: (was: hadoop-15142.002.patch) > Register FTP and SFTP as FS services > > > Key: HADOOP-15142 > URL: https://issues.apache.org/jira/browse/HADOOP-15142 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.0.0 >Reporter: Mario Molina >Priority: Minor > Fix For: 3.0.1 > > Attachments: HADOOP-15142.001.patch > > > SFTPFileSystem and FTPFileSystem are not registered as a FS services. > When calling the 'get' or 'newInstance' methods of the FileSystem class, the > FS instance cannot be created due to the schema is not registered as a > service FS. > Also, the SFTPFileSystem class doesn't have the getScheme method implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15027: --- Status: Patch Available (was: Open) > AliyunOSS: Support multi-thread pre-read to improve sequential read from > Hadoop to Aliyun OSS performance > - > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch, HADOOP-15027.010.patch, HADOOP-15027.011.patch, > HADOOP-15027.012.patch, HADOOP-15027.013.patch, HADOOP-15027.014.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15027: --- Status: Open (was: Patch Available) To trigger the build which should happen in place. > AliyunOSS: Support multi-thread pre-read to improve sequential read from > Hadoop to Aliyun OSS performance > - > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch, HADOOP-15027.010.patch, HADOOP-15027.011.patch, > HADOOP-15027.012.patch, HADOOP-15027.013.patch, HADOOP-15027.014.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15027: --- Summary: AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance (was: AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance) > AliyunOSS: Support multi-thread pre-read to improve sequential read from > Hadoop to Aliyun OSS performance > - > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch, HADOOP-15027.010.patch, HADOOP-15027.011.patch, > HADOOP-15027.012.patch, HADOOP-15027.013.patch, HADOOP-15027.014.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328266#comment-16328266
]
genericqa commented on HADOOP-14707:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 29m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 39s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
37s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
5s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
48s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
54s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
35s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestFilterFileSystem |
| | hadoop.fs.TestHarFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14707 |
| JIRA Patch URL |
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328262#comment-16328262 ] SammiChen commented on HADOOP-15027: My + 1 to the patch. Will commit after [~genericqa] comment comes out. Thanks [~wujinhu] for the contribution. > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch, HADOOP-15027.010.patch, HADOOP-15027.011.patch, > HADOOP-15027.012.patch, HADOOP-15027.013.patch, HADOOP-15027.014.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15168) Add kdiag tool to hadoop command
[
https://issues.apache.org/jira/browse/HADOOP-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328207#comment-16328207
]
genericqa commented on HADOOP-15168:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
8s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
9s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15168 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906268/HADOOP-15168.00.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 8d2c7fefa1d3 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 41049ba |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13974/testReport/ |
| Max. process+thread count | 312 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13974/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add kdiag tool to hadoop command
>
>
> Key: HADOOP-15168
> URL: https://issues.apache.org/jira/browse/HADOOP-15168
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15168.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328031#comment-16328031 ] Yufei Gu commented on HADOOP-15157: --- Hi [~grepas], do you need it on 3.0.x or branch-2? > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Fix For: 3.1.0 > > Attachments: HADOOP-15157.000.patch, HADOOP-15157.001.patch, > HADOOP-15157.002.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the hadoop.zk.auth and ha.zookeeper.auth properties > can be retrieved via the CredentialProviderAPI that's been configured using > the credential.provider.path, with fallback provided to the clear-text value > or @/path/to/file notation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328003#comment-16328003 ] Aaron Fabbri commented on HADOOP-13761: --- [~ste...@apache.org] sure. > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Steve Loughran >Priority: Blocker > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327943#comment-16327943 ] Konstantin Shvachko commented on HADOOP-12751: -- Yes it should be fine to back-port this to branch-2.7. > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 > Environment: kerberos >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Labels: kerberos > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0001-Remove-check-for-user-name-characters-and.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch, > 0005-HADOOP-12751-leave-user-validation-to-os.patch, > 0006-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15168) Add kdiag tool to hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-15168: Attachment: HADOOP-15168.00.patch > Add kdiag tool to hadoop command > > > Key: HADOOP-15168 > URL: https://issues.apache.org/jira/browse/HADOOP-15168 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HADOOP-15168.00.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15168) Add kdiag tool to hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-15168: Attachment: (was: HADOOP-15168.00.patch) > Add kdiag tool to hadoop command > > > Key: HADOOP-15168 > URL: https://issues.apache.org/jira/browse/HADOOP-15168 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15168) Add kdiag tool to hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-15168: Attachment: HADOOP-15168.00.patch > Add kdiag tool to hadoop command > > > Key: HADOOP-15168 > URL: https://issues.apache.org/jira/browse/HADOOP-15168 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HADOOP-15168.00.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15168) Add kdiag tool to hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-15168: Status: Patch Available (was: Open) > Add kdiag tool to hadoop command > > > Key: HADOOP-15168 > URL: https://issues.apache.org/jira/browse/HADOOP-15168 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HADOOP-15168.00.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15168) Add kdiag tool to hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HADOOP-15168: Summary: Add kdiag tool to hadoop command (was: Add kdiag and HadoopKerberosName tools to hadoop command) > Add kdiag tool to hadoop command > > > Key: HADOOP-15168 > URL: https://issues.apache.org/jira/browse/HADOOP-15168 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14707:
Status: Patch Available (was: Open)
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch
>
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327560#comment-16327560 ] Hudson commented on HADOOP-15157: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13501 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13501/]) HADOOP-15157. Zookeeper authentication related properties to support (yufei: rev b2029353537fc8da9ab67834568cb2e24924cf5a) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md * (edit) hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithNFS.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Fix For: 3.1.0 > > Attachments: HADOOP-15157.000.patch, HADOOP-15157.001.patch, > HADOOP-15157.002.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the hadoop.zk.auth and ha.zookeeper.auth properties > can be retrieved via the CredentialProviderAPI that's been configured using > the credential.provider.path, with fallback provided to the clear-text value > or @/path/to/file notation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15062) TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
[
https://issues.apache.org/jira/browse/HADOOP-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated HADOOP-15062:
--
Status: Patch Available (was: Open)
> TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
> ---
>
> Key: HADOOP-15062
> URL: https://issues.apache.org/jira/browse/HADOOP-15062
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15062.000.patch
>
>
> {code}
> [ERROR]
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Time
> elapsed: 0.478 s <<< FAILURE!
> java.lang.AssertionError: Unable to instantiate codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, is the required version of
> OpenSSL installed?
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec.init(TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java:43)
> {code}
> This happened due to the following openssl change:
> https://github.com/openssl/openssl/commit/ff4b7fafb315df5f8374e9b50c302460e068f188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15062) TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
[
https://issues.apache.org/jira/browse/HADOOP-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327521#comment-16327521
]
Yufei Gu commented on HADOOP-15062:
---
Submit to kick off the Jenkins.
> TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
> ---
>
> Key: HADOOP-15062
> URL: https://issues.apache.org/jira/browse/HADOOP-15062
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15062.000.patch
>
>
> {code}
> [ERROR]
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Time
> elapsed: 0.478 s <<< FAILURE!
> java.lang.AssertionError: Unable to instantiate codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, is the required version of
> OpenSSL installed?
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec.init(TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java:43)
> {code}
> This happened due to the following openssl change:
> https://github.com/openssl/openssl/commit/ff4b7fafb315df5f8374e9b50c302460e068f188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated HADOOP-15157: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Fix For: 3.1.0 > > Attachments: HADOOP-15157.000.patch, HADOOP-15157.001.patch, > HADOOP-15157.002.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the hadoop.zk.auth and ha.zookeeper.auth properties > can be retrieved via the CredentialProviderAPI that's been configured using > the credential.provider.path, with fallback provided to the clear-text value > or @/path/to/file notation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327511#comment-16327511 ] Yufei Gu commented on HADOOP-15157: --- Committed to trunk. Thanks for the patch [~grepas]! Thanks for the review [~lmccay]. > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Attachments: HADOOP-15157.000.patch, HADOOP-15157.001.patch, > HADOOP-15157.002.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the hadoop.zk.auth and ha.zookeeper.auth properties > can be retrieved via the CredentialProviderAPI that's been configured using > the credential.provider.path, with fallback provided to the clear-text value > or @/path/to/file notation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15151) MapFile.fix creates a wrong index file in case of block-compressed data file.
[ https://issues.apache.org/jira/browse/HADOOP-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327361#comment-16327361 ] Grigori Rybkine commented on HADOOP-15151: -- Can this patch be reviewed, please? (The latest test failures are not relevant to the patch.) Thanks. > MapFile.fix creates a wrong index file in case of block-compressed data file. > - > > Key: HADOOP-15151 > URL: https://issues.apache.org/jira/browse/HADOOP-15151 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Grigori Rybkine >Priority: Major > Labels: patch > Attachments: HADOOP-15151.001.patch, HADOOP-15151.002.patch, > HADOOP-15151.003.patch, HADOOP-15151.004.patch > > > Index file created with MapFile.fix for an ordered block-compressed data file > does not allow to find values for keys existing in the data file via the > MapFile.get method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Status: Patch Available (was: Open)
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15167) [viewfs] ls will fail when user doesn't exist
[
https://issues.apache.org/jira/browse/HADOOP-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326976#comment-16326976
]
Vinayakumar B commented on HADOOP-15167:
In general for HDFS, primary group of a user is not required. Owner group for a
file/directory is inherited from parent, unless set explictly using setOwner()
with non-empty group. So FileStatus for HDFS files will always contain group
name. It will not be empty AFAIK.
So in this particular case of ViewFileSystem, primary group is used to create
the FileStatus object for internal read-only mount table inodes, which are just
local to this instance of FileSystem instance.
So using the name of the user itself as group (in case of no primary group)
makes sense.
FileStatus code can handle null/empty group/owner. Not a problem from code
perspective.
W.r.t compatibility,
If group is empty, '*ls*' command prints just space in place of group. This
breaks the parsers which depends on 'ls' command output.
> [viewfs] ls will fail when user doesn't exist
> -
>
> Key: HADOOP-15167
> URL: https://issues.apache.org/jira/browse/HADOOP-15167
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15167.patch
>
>
> Have Secure federated cluster with atleast two nameservices
> Configure viewfs related configs
> When we run the {{ls}} cmd in HDFS client ,we will call the method:
> org.apache.hadoop.fs.viewfs.ViewFileSystem.InternalDirOfViewFs#getFileStatus
> it will try to get the group of the kerberos user. If the node has not this
> user, it fails.
> Throws the following and exits.UserGroupInformation#getPrimaryGroupName
> {code}
> if (groups.isEmpty()) {
> throw new IOException("There is no primary group for UGI " + this);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-14999:
---
Attachment: HADOOP-14999.005.patch
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Major
> Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch,
> HADOOP-14999.003.patch, HADOOP-14999.004.patch, HADOOP-14999.005.patch,
> asynchronous_file_uploading.pdf
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
> This patch reuse {{SemaphoredDelegatingExecutor}} as executor service and
> depends on HADOOP-15039.
> Attached {{asynchronous_file_uploading.pdf}} illustrated the difference
> between previous {{AliyunOSSOutputStream}} and
> {{AliyunOSSBlockOutputStream}}, i.e. this asynchronous multi-part based
> uploading mechanism.
> 1. {{AliyunOSSOutputStream}}: we need to output the whole result to local
> disk before we can upload it to OSS. This will poses two problems：
> - if the output file is too large, it will run out of the local disk.
> - if the output file is too large, task will wait long time to upload
> result to OSS before finish, wasting much compute resource.
> 2. {{AliyunOSSBlockOutputStream}}: we cut the task output into small blocks,
> i.e. some small local file, and each block will be packaged into a uploading
> task. These tasks will be submitted into {{SemaphoredDelegatingExecutor}}.
> {{SemaphoredDelegatingExecutor}} will upload this blocks in parallel, this
> will improve performance greatly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-14999:
---
Attachment: (was: HADOOP-14999.005.patch)
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Major
> Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch,
> HADOOP-14999.003.patch, HADOOP-14999.004.patch, HADOOP-14999.005.patch,
> asynchronous_file_uploading.pdf
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
> This patch reuse {{SemaphoredDelegatingExecutor}} as executor service and
> depends on HADOOP-15039.
> Attached {{asynchronous_file_uploading.pdf}} illustrated the difference
> between previous {{AliyunOSSOutputStream}} and
> {{AliyunOSSBlockOutputStream}}, i.e. this asynchronous multi-part based
> uploading mechanism.
> 1. {{AliyunOSSOutputStream}}: we need to output the whole result to local
> disk before we can upload it to OSS. This will poses two problems：
> - if the output file is too large, it will run out of the local disk.
> - if the output file is too large, task will wait long time to upload
> result to OSS before finish, wasting much compute resource.
> 2. {{AliyunOSSBlockOutputStream}}: we cut the task output into small blocks,
> i.e. some small local file, and each block will be packaged into a uploading
> task. These tasks will be submitted into {{SemaphoredDelegatingExecutor}}.
> {{SemaphoredDelegatingExecutor}} will upload this blocks in parallel, this
> will improve performance greatly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326950#comment-16326950 ] Vinayakumar B commented on HADOOP-12751: This looks to be good candidate for branch-2.7. Do you agree [~shv] ? Not a direct cherry-pick. Patch needs update for branch-2.7 > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 > Environment: kerberos >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Labels: kerberos > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0001-Remove-check-for-user-name-characters-and.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch, > 0005-HADOOP-12751-leave-user-validation-to-os.patch, > 0006-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Attachment: HADOOP-15027.014.patch > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch, HADOOP-15027.010.patch, HADOOP-15027.011.patch, > HADOOP-15027.012.patch, HADOOP-15027.013.patch, HADOOP-15027.014.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
